Define JSON representation of the Elm AST

[avh4]

This github issue will be used to brainstorm and discuss a JSON representation of the Elm AST. The goal is for a future version of elm-format to be able to convert between Elm source code and an AST representation in JSON, which will make it easier for other tools to be written that inspect or manipulate Elm code.

[ajhyndman]

I'd like to help by trying to document some of the existing AST structure. Is anyone able to help point me in the right direction to start exploring the current functionality?

[avh4]

The current AST is defined here: https://github.com/avh4/elm-format/tree/master/parser/src/AST

Some considerations: we may want to rename some things internally to match the names that a user of Elm would be familiar with.

There is also this previous attempt at formalizing Elm syntax: https://github.com/Apanatshka/elm-spoofax

A good first step would be to see if/how other languages provide a structured representation of their AST to see if there is anything to be learned about what works well.

In general, I would like to define a good JSON AST representation first, and them implement it, rather than simply rendering the current internal AST into JSON.

[rundis]

Here's the scala one : http://www.scala-lang.org/files/archive/spec/2.11/13-syntax-summary.html

[rundis]

I'm not sure if this is useful, but this is the current definition for the AST I'm using in elm-light version 0.4.0. http://peg.arcanis.fr/3bbplj/1/

(it's by no means complete and probably has quite a few errors), but maybe it gives some ideas. I'm only using a small percentage of the nodes and I'd probably structure things quite differently once I start using more of the info.

I was and still am reluctant to make this public, because I don't really think it's a good idea to encourage others to use this stuff. People should rather help out to make the elm-format AST the de-facto thing to use imho.

If this isn't really useful at all, let me know and I'll delete this comment :-)

[Janiczek]

Relevant work: https://github.com/vilterp/elm-tracing-interpreter

[avh4]

Other AST examples to look at:

• Elixir AST
• gcc XML output
• Esprima (javascript)

[gyzerok]

@avh4 probably a good idea would be to look at Babel AST. There are insane amount of tools build on top of it, so hopefully they've done a great job in making AST usability a great experience.

[georgemarrows]

@gyzerok by "Babel AST", do you mean this?

This is good for reviewing various Javascript AST styles: https://astexplorer.net/

[knewter]

I would like to build a project to consume an Elm AST for a TEA and produce code for an Elixir GenServer and a corresponding Phoenix Channel to make a trivial realtime starting point between Phoenix and Elm. I've done it by hand and the result was pleasant. I want more of these in the wild :)

[zkessin]

I would like to be abe to build a tool that could find all of the commands that would create an ajax call to a server and then validate that the data being sent across the channel is of the right shape.

So for example lets say I have an app in elm and a backend in Erlang, Elixir, Haskell, Ruby etc. I want to create a quickcheck test on the server that says "ELM will be sending values like this" and Expecting values like that back, and have it be tested somehow.

[zkessin]

It would be good to provide and ELM ast as well typed Elm Data. That way you could work with it in elm.

WhatI would like to see is a set of functions that would have types like this

compile: ElmSource -> Result ElmAst
decompile: ElmAst -> Result ElmSource

(With ElmSource really just being String)

That way we can write a set of transforms of the form
ElmAst -> Result ElmAst that could do things like refactoring, validations, etc.

[avh4]

Tonight I set up the skeleton for the JSON AST command line option and for having tests for the AST output. You can follow along here: https://github.com/avh4/elm-format/blob/json/tests/test-files/good/AllSyntax/0.16/Expressions.json

I'm already running in to lots of cases where the public AST should be different from the internal AST; here are a couple of them:

• top-level definitions can technically have the left-most term be any pattern destructuring, but in practice this is rare; it seems ideal to handle the common case as a special AST node type so that tools won't have to dig into the pattern AST to figure out the names of top-level declarations
• it will also probably be ideal to combine the top-level declarations and the type annotations into a single AST node rather than having them be separate

[rundis]

it will also probably be ideal to combine the top-level declarations and the type annotations into a single AST node rather than having them be separate

Absolutely. That's the approach I took for my parser in elm-light and it made quite a few things more convenient. It would probably also make sense to have the doc string (if any) in the same node too.

[zkessin]

Just make sure that everything has a file and line number attached to it when you build the AST, because if you are using the AST for any kind of validation or testing it will be really handy to know that this clause came from Foo.elm on line 41

[michaeljones]

Just coming across this. I have been thinking about this a little with the motivation being around generating Elm code easily. Particularly generating functions from ICU internationalisation strings. I felt that it would be easier to programmatically generate an AST as Json or something and then have a central tool that writes the Elm code from the AST so I don't have to worry about getting the indentation correct and all that. So a fairly standard use case, but just wanted to share.

[mbylstra]

Hi, I wondering if the goal was to generate a "canoncial AST" (https://github.com/elm/compiler/blob/master/compiler/src/AST/Canonical.hs) or just the AST for a single file? (An AST that has no understanding of where variables are defined). Knowledge of where variables are defined does not appear to be necessary for doing code formatting, but I'm wondering if there are plans for this anyway. The only use case for a code formatter I can think of is grouping imports that come from imported packages vs those defined locally. My particular use case (generating dummy data for arbitrary type signatures) requires knowing where variables are defined if they are defined in other modules. I wonder if generating a canonical AST would be heading too far into language server terrority (does not seem to be a goal for this tool), and would overcomplicate this tool intended for fast code formatting.

I'm guessing that this public AST issue hasn't seen much activity since 2016 as everyone has been waiting on 0.19 before making the next move. Now that 0.19 is out, are there still plans for the public AST version? I'm trying to decide if I should wait and take advantage of this tool, or go ahead and make my own fork of the 0.19 compiler for my own purposes. I'm also familiar with https://github.com/stil4m/elm-analyse and https://github.com/stil4m/elm-syntax which could also be used, but I prefer the fork-the-elm-compiler Haskell approach.