Path to astexplorer v3

[fkling]

The Problem

astexplorer provides access to variety of parsers and transform tools. This is done rather naively by bundling everything together as one website. We do code splitting, but UI code is still depending on parser and transformer code. There are a couple of issues with that approach:

• Updating a parser means deploying a new version of the whole site
• Building the site takes time (it's OK, but not great)
• Confusing dependencies: eslint is both transformer and dev-dependency.

More issues with the architecture of astexplorer:

• Parser output (AST) and transformer are disconnected. The settings your provide to the parser have no impact on the transformer.
• There can only exist one version of a parser/transformer. Multiple versions can be provided but with some difficulty.

The Goal

Instead of bundling parsers/transformers with the website, each parser/transformer will be built in isolation and loaded as a standalone bundle. This provides the following advantages:

• Parsers/transformers are decoupled from each other, reducing potential conflicts.
• The UI is completely decoupled from parsers/transformers. Updating either doesn't require re-bundling the other.
• We can update parsers/transformers automatically.

In addition, instead of loading a parser that is completely unconnected to the selected transformer, transformers will expose a parse method that delegates to the transformer's own parser (instead of just suggesting which parser it should be used with).

I.e. parser and transformers now live "in the same space". Parser have only a parse method. Transformers have parse and transform methods.

Next Steps

As so often, I wanted to implement everything at once, but doing this incrementally is more manageable.

• Refactor parser/transform API to isolate them better.
• Update transform definitions to expose parse methods.
  Easy if the transformer already exposes it itself. Otherwise we have to reach into its internals (ugh), or replicate its internal parsing logic (also kind of ugh).
• Update menus so either a parser or a transform can be selected, but not both.
• Update storage classes to either store parser or transformer IDs, but not both.
• Generate/write webpack configs for every parser/transform. Generalize it so that different versions of the same parser/transform can be built.
• Setup cronjob to build the latest versions and manifest file (for parser selection).
• Add server endpoint to load the given version of the parser/transform.
• Update UI to use manifest and parser/transformer loader endpoint

Note: I cannot make any promises regarding the time frame of this, I'm working on it as I can.

[0xdevalias]

Curious if there are any updates on the v3 plan over the last ~5 years or so?

See also:

• State of the Project #625
• Make astexplorer standalone / embeddable  #70
• Any plans to encapsulate parsers as NPM package? #646
• Feature request: Tree of JavaScript scopes (eslint-scope) #510
• Show JSON path #310