WIP: Use new parser API

[c42f]

These changes allow CSTParser to be connected up as the main Julia parser.

To use it, you currently need both a branch of the Julia runtime and of CSTParser:

• Replaceable single entry point for parser julia#35243
• https://github.com/c42f/CSTParser.jl/tree/cjf/Expr-linenumbers

Then run FancyDiagnostics.enable!(), and we can have errors such as:

Dev notes

Early screenshot:

Note that the current implementation is pretty rough and diagnostic printing will still crash at the drop of a hat (I still need to understand how CSTParser uses ErrorToken a lot better and figure how to cleanly deal with offsets at EOF which are very common)

TODO:

• Better understand how ErrorToken is used to encode the span, and what the deal is with nested error tokens.
• Make handling errors at EOF actually work
• Detect incomplete expressions and return Expr(:incomplete)
• Much better error messages. The above example is not really representative in that CSTParser has only a handful of very generic error states. (Perhaps the CST itself will provide the context for more useful messages, or perhaps we'll need to put messages back into CSTParser itself.)

[omus]

You are probably aware but I'll mention there is printstyled(io, str, color=:underline)

[c42f]

Yes that's an interesting option - perhaps worth doing, especially for multi-line errors. We could use underline, red,bold or some other styling. (It's even an option to have syntax highlighting combined with error reporting - Tokenize + CSTParser gives us more than enough tooling for that.)

The thing which I find nice about the box chars is that they work in black and white, and they can be colored separately from the source code. But the need to format multi-line errors may override that consideration.

[c42f]

As you can see in the screencast, I've tried in-situ highlighting in the latest version which is an easier option when the error spans multiple lines.