Reworking some Julia AST forms

[c42f]

With JuliaSyntax, we've got our own green tree (GreenNode) and AST (SyntaxNode) which often differ from Expr, due to the requirement that children are strictly in source order. Some current differences are described in https://github.com/JuliaLang/JuliaSyntax.jl#tree-differences-between-greennode-and-expr. Given that we've been forced to diverge we might as well make the most of this and reconsider some aspects of Expr for two reasons:

• To give extra textural information to users of green tree (eg, presence of parentheses), without the need to recompute this information by inspecting the trivia. (These nodes will probably be elided during AST conversion?)
• To give extra information to AST users like macro writers (eg, conditional ternary is the same as :if for Expr users)

List of possible changes

• Insertion of extra blocks inconsistent in various places, sometimes due to the need of Expr to hold LineNumberNode. Avoid this!
  • let argument lists (Always emit a block for let binding lists #126)
  • short form functions have a block for the body line number
  • elseif conditional has a block for the line number
  • anon function -> has a block only for the body line number
• Containers for string-like constructs - these are very useful for tooling which works with the tokens of the source text. Also separating the delimiters from the text they delimit removes a whole class of tokenization errors and lets the parser deal with them.
  • string (always use K"string" to wrap strings, even when they only contain a single K"String" token. See Always use string nodes wrapping string chunks #94)
  • char (add a new K"char" for containing characters along with their delimiters?). See Split char delimiters early and emit K"char" node #121
  • backticks (use K""or addK"backticks"`?)
  • var"" syntax (use K"var"? as the head?) (Wrap var"" nonstandard identifiers in var nodes #127)
• Faithful representation of the source text / avoid premature lowering
  • K"macrocall" - allow users to easily distinguish macrocalls with parentheses from those without them (Add PARENS_FLAG to tuple, block and macrocall #218)
  • parentheses (K"parens")
  • Ternary conditional (Use K"?" for head of ternary conditional #85)
  • global const is normalized to const global in the parser. Should be done in Expr conversion (Fix const struct field errors + cleanup global const AST #130)
  • The AST for do seems partially lowered. Should it be flatter like f(x) do y ; body end being (do (call f) (tuple y) (block body))? (Parse do syntax without desugaring the closure #98)
  • Lower @. to @__dot__ later, not inside parser
  • Lower docstrings to Core.@doc later, not inside parser (Parse docstring attachment as K"doc" kind #217)
• Improvements for AST inconsistencies
  • The AST for dotted infix calls like a .+ b is inconsistent with prefix calls like f.(a, b) (see Rework handling of . tokenization and dotted operator calls #90)
  • "Standalone" dotted operators are parsed inconsistently. For example .*(x,y) ==> (call .* x y) vs (.*)(x,y) ==> (call (. *) x y) ie, there's a dotted symbol vs Expr(:.) in these cases (see discussion in Rework handling of . tokenization and dotted operator calls #90)
  • Inconsistent use of = vs kw (AST inconsistency between parsing of = vs kw  #99)
  • Emit unadorned postfix adjoint with a call head rather than as a syntactic operator for consistency with suffixed versions like x'ᵀ (Record fixity of call type in flags #124 )
• Improvements to awkward AST forms
  • Frakentuples with multiple parameter blocks like (a=1, b=2; c=3; d=4) are a weird nested structure. Let's flatten this and bring the multiple sets of `K"parameters" into the parent tuple. (More flattened form for multiple frakentuple parameters #133)
  • try-catch-else-finally is very awkward with its optional children. Is there something we can/should do about this? How about lowering try body1 catch exc body2 else body3 finally body4 end to (try body1 (catch exc body2) (else body3) (finally body4)) (Simplify child layout for try #234)
  • Import paths import A.b.c are different from normal A.b.c. Use a different head for these.
  • Some constructs have a first child which serves as a flag, for example whether mutable is present in a struct and whether a module is actually baremodule. These seem more like part of the expression head - would they be better as flags?

Considered and rejected for now

• :toplevel expressions occur both at file scope and as ;-delimited expressions at file scope. This seems kind of weird?
• Is it possible to improve the representation of global x,y vs global x vs global (x,y) = (1,2) vs global (x,y)? (In particular, the variables might or might not be enclosed in a tuple.)
• Lower cmd strings to Core.@cmd later, not inside parser
• Lower custom string and cmd macros to macrocall later, not inside parser
• Infix vs prefix calls - should this be managed in flags or a kind like K"infix_call"? (considered and decided against in AST inconsistency between parsing of = vs kw  #99 and Record fixity of call type in flags #124)

[c42f]

I think this is done for now. It's become quite an epic list of changes 😬

There might be more but they can go in separate issues I guess.

[timholy]

Seems like a considerable increase in consistency!

[c42f]

Thanks I hope so! It added up to a lot of rearrangement :-)