Babel compatible AST

[Gregoor]

Heya,

thanks for this amazingly fast parser! I've been playing with replacing it with Babel's parser, but I also used a number of babel libraries which expect a Babel AST. Namely span to be in-lined (collapsing LogicalExpression into BinaryExpression does not seem to be causing problems). Now I was wondering whether the divergence from Babel's AST is wanted? I can imagine integration into all kinds of related libs to be simpler when conforming to their AST more. Or is there a perf cost associated with these differences?
Would be curious to learn more and I did not find a past issues that made me understand.

Cheers!

Edit: Oh I also just saw that BlockStatement is also shaped differently

[kdy1]

It's not hard, but not trivial at all.
It would require really lots of work.

[dsherret]

There are also some cases were babel made some incorrect namings in their AST #548

Also, generally SWC moves faster than Babel and so sometimes SWC is ahead with their support for TS language features. So SWC could implement something then Babel could come along and implement it with a different AST shape.

[dwoznicki]

Just curious, won't this eventually need to be done to support #1321 ?

[kdy1]

@dwoznicki Yes, this is required for #1321.

[coderaiser]

Thank you for such a nice project :). I'm working on putout code transformer, it has support of the most popular JavaScript parsers that producing estree and babel AST (with help of estree-to-babel adapter). This made to have ability to switch into fastest one, right now babel is default.

As I see swc should be the fastest parser, because it's written on Rust and as I understood it's willing to be successor of babel. I think that would be amazing to add support of swc parser (or switch to it in the future, if it really will be faster) to putout, but I see that it has different AST format. Would be great to see all deviations from babel AST, as I understood from @Gregoor they are not to many, so some adapter can be written for this purpose (hope it wan't kill all speed :)).

About TS language it's not a problem that syntax is different, anyways would be great to have a list of differences, to simplify switching from one tool to another. Also babel has a lot plugins, reach ecosystem and even a book, so would be beneficial to have ability to reuse all this tools that using Babel AST :).

[kdy1]

@coderaiser I know it worths it, but I just don't have time.
And as swc is also used from rust side, changing ast definitions to match one of babel is unacceptable.
swc-to-babel, written in rust, would be a better solution and would not be hard.

[coderaiser]

@kdy1, swc-to-babel adapter is great, even written in JavaScript, but in Rust it will be definitely faster.

Also would be amazing to have ability to get tokens list from the root of ast. It needed for recast, that helps to change only code that matters, not the whole file. In other case another parser should be used to get tokens list (esprima by default) and this can nullify all speed benefits of swc.

[Gregoor]

Heya, I noticed babelify is still commented out in the exported JS, are there issues you consider blocking before exposing it?

[kdy1]

Yes, some more works are required

[kdy1]

I started working on babelify and swcify.
I'd like to know your usecase to determine api. (e.g. file vs ast vs source code for input)
Can you tell me about it?

[kdy1]

I'm also going to write babel-plugin runner so I'm not sure if babelify is necessary.

[coderaiser]

I started working on babelify and swcify.
I'd like to know your usecase to determine api. (e.g. file vs ast vs source code for input)
Can you tell me about it?

I want to use swc as drop-in replacement of babel parser in putout linter, to use as an input for recast.

Also would be nice to have generator, but parser is also amazing, especially if it will be faster.

[yisar]

Maybe acorn's AST is more standard.
https://astexplorer.net/

[kdy1]

I tried this, but actually, it was slower.
Parsing as babel AST was performant enough, even though it does parsing and converting.

But returning the parsed ast back to javascript world was too slow.

This is the number I get.

This is rust-side benchmark, which uses same file but from rust side.

It's about 37365 ops / sec.

I'll investigate a way to make crossing boundaries faster.

[SoloJiang]

In my mind, babelify may not the most important task. The target may make the js plugin system more user friendly. For example: 1. create some type node faster; 2. node type checker; 3. playground; 4. let AST structure to code like babel/generator ...

[kdy1]

@SoloJiang
For 1, almost all of slowdown comes from passing ast nodes from rust to node.
(It might be worth using wasm, as V8 is free to optimize it while native addons are not)

For 2, I'm working hard on it.

For 3, https://swc.rs/docs/online-repl although it's does not show ast.

For 4, can you elaborate? What does code like babel/generator means?

[SoloJiang]

For 4: like this https://babeljs.io/docs/en/babel-generator. Developer can use it verify their plugin quickly.

[SoloJiang]

For 1: my meaning is not runtime speed. Okay, my presentation problem. I mean may it create AST nodes very quickly....

[yisar]

@SoloJiang @kdy1

I don't think it's necessary to be compatible with Babel's AST standard, but I'm still worried about the data transfer between rust and js, which is a performance bottleneck. I have seen that parcel operates AST directly on the rust side (using wasm), which seems to be the best solution at present, but swc does not provide a plugin system for rust.

[kdy1]

I'll fix swc.print() along while fixing the plugin apis.
(I need to verify codegen anyway to verify plugin api.

[SoloJiang]

@yisar you are right, but js plugin system for the front end developer is more friendly? rust plugin system and js plugin system, we all need =。= ... The decision to use the plugin system should be the developer's.

[kdy1]

@yisar I came up with an idea. I didn't make rust plugin system because

• rust does not have abi and the ast definitions of swc are changing quite frequently
• deploying node binaries are hard

But passing down ast as json to plugin would be performant enough and I can create a wrapper for swc plugins using proc macros.

@SoloJiang Do you mean users of swc should decide the language used for plugins?

[SoloJiang]

For 1: my meaning is not runtime speed. Okay, my presentation problem. I mean may it create AST nodes very quickly....

For this, I can create import xxx from 'ABC' by:

t.importDeclaration([t.importDefaultSpecifier(t.identifier(localName)], t.stringLiteral(value));

[SoloJiang]

@kdy1 Yeah, I mean this. For performance requirements or for parsing too many files, I might choose Rust. Select JS for other scenarios

[SoloJiang]

For 3: I mean AST playground, hhhhh

[yisar]

@kdy1 Unfortunately, I have tested the performance of swc_wasm, but it is still very slow.

I feel that it is unreliable to operate AST in js side, because there is no good way to optimize it by serializing AST as JSON and passing it back and forth.

Now I can only use swc to operate AST in rust side, and then pass a simple structure to js side.

[kdy1]

@yisar Yes, I also think so.
I'll provide rust plugin API and plugins for various open-source projects.

Porting a babel plugin is very easy task to me, but API for plugins is the hard part.

[jdalton]

Just tying this issue to #1366 as any adapter will need valid span/range/start/end values (which are lacking at the moment)

[StJohn3D]

Hi there, just heard of SWC today and I'm not sure if this is the right place to ask but - I'm curious how one would go about migrating their babel macros to work with SWC ?? I see some documentation on plugins but doesn't quite look like the same thing.
Most common problem trying to solve is reading in values from package.json or environment variables at compile time. But the @emotion/styled/macro would also be a huge win / non-start to switching away from babel.

[kdy1]

I'm going to provide a rust plugin api and deprecate node apis.

Babel plugins are not supported.

[jdb8]

I'm going to provide a rust plugin api and deprecate node apis.

Just to confirm, does this mean that we should avoid writing plugins similar to the example here: https://swc.rs/docs/usage-plugin? Will this kind of js plugin be deprecated in the future in favour of rust-only plugins?

[kdy1]

@jdb8 Correct. It's simply misdesigned.

[jdb8]

@kdy1 good to know, thanks - I wonder if it would be worth updating that page with a deprecation warning? I was briefly considering porting some plugins from Babel to swc, but I'll definitely now wait until the Rust api is finalised.

[kdy1]

I added it. Thanks!

[StJohn3D]

@kdy1 would this rust plugin api allow us to build macros similar to babel macros?

[kdy1]

Yes it will.

[overlookmotel]

I think #2175 is relevant to this discussion. I've done some experiments into ways to speed up parse().

[Gregoor]

Out of curiosity, with the Node API deprioritized, is Babel AST compatibility still of interest to this project?

[kdy1]

@Gregoor If there's a need that is helpful to the project, yes.

[coderaiser]

This can be a solution: swc-to-babel. But I need a comments as well. I'm working on 🐊Putout, and comments really needed 🤷‍♂️..

Everything else can be converted, but that's just missing information...

Here is AST examples for Babel:

{
  "type": "File",
  "start": 0,
  "end": 29,
  "loc": {
    "start": {
      "line": 1,
      "column": 0
    },
    "end": {
      "line": 3,
      "column": 1
    }
  },
  "errors": [],
  "program": {
    "type": "Program",
    "start": 0,
    "end": 29,
    "loc": {
      "start": {
        "line": 1,
        "column": 0
      },
      "end": {
        "line": 3,
        "column": 1
      }
    },
    "sourceType": "module",
    "interpreter": null,
    "body": [
      {
        "type": "FunctionDeclaration",
        "start": 9,
        "end": 29,
        "loc": {
          "start": {
            "line": 2,
            "column": 0
          },
          "end": {
            "line": 3,
            "column": 1
          }
        },
        "id": {
          "type": "Identifier",
          "start": 18,
          "end": 23,
          "loc": {
            "start": {
              "line": 2,
              "column": 9
            },
            "end": {
              "line": 2,
              "column": 14
            },
            "identifierName": "world"
          },
          "name": "world"
        },
        "generator": false,
        "async": false,
        "params": [],
        "body": {
          "type": "BlockStatement",
          "start": 26,
          "end": 29,
          "loc": {
            "start": {
              "line": 2,
              "column": 17
            },
            "end": {
              "line": 3,
              "column": 1
            }
          },
          "body": [],
          "directives": []
        },
        "leadingComments": [
          {
            "type": "CommentLine",
            "value": " hello",
            "start": 0,
            "end": 8,
            "loc": {
              "start": {
                "line": 1,
                "column": 0
              },
              "end": {
                "line": 1,
                "column": 8
              }
            }
          }
        ]
      }
    ],
    "directives": []
  },
  "comments": [
    {
      "type": "CommentLine",
      "value": " hello",
      "start": 0,
      "end": 8,
      "loc": {
        "start": {
          "line": 1,
          "column": 0
        },
        "end": {
          "line": 1,
          "column": 8
        }
      }
    }
  ]
}

And SWC:

{
  "type": "Module",
  "span": {
    "start": 0,
    "end": 12,
    "ctxt": 0
  },
  "body": [
    {
      "type": "VariableDeclaration",
      "span": {
        "start": 0,
        "end": 12,
        "ctxt": 0
      },
      "kind": "const",
      "declare": false,
      "declarations": [
        {
          "type": "VariableDeclarator",
          "span": {
            "start": 6,
            "end": 11,
            "ctxt": 0
          },
          "id": {
            "type": "Identifier",
            "span": {
              "start": 6,
              "end": 7,
              "ctxt": 0
            },
            "value": "a",
            "optional": false,
            "typeAnnotation": null
          },
          "init": {
            "type": "NumericLiteral",
            "span": {
              "start": 10,
              "end": 11,
              "ctxt": 0
            },
            "value": 5
          },
          "definite": false
        }
      ]
    }
  ],
  "interpreter": null
}