Rewriting Lebab as a Babel transformer? #138

Open
mohebifar opened this Issue Jun 11, 2016 · 16 comments

Projects

None yet

4 participants

@mohebifar
Member

Currently lebab is doing so much from parsing to generating code. During the process, somethings are missing (Like nice warnings while parsing the code #136 or JSX Support #130) or even some bugs show up. I believe almost all of these issues have been considered and solved in babel.

Lebab's aim is basically transforming an old code to a modern code. That means the only thing that matters in lebab is transforming. So, making a babel transformer instead of re-implementing all that stuff can be an option for the future releases.

@nene
Collaborator
nene commented Jun 11, 2016

That's an intriguing idea, but I to my understanding Babel does not fully preserve the original whitespace. For example, when transforming the following code, that has nothing to transform:

/**
 * My func
 */
function  foo   (y) {
    // Hello world
    if (  true  ) {
            // poor indentation
    }
}

Babel produces:

/**
 * My func
 */
function foo(y) {
  // Hello world
  if (true) {
    // poor indentation
  }
}

It does preserve all the comments, but it pretty much reformats all the whitespace. This behavior is fine for Babel, as one only cares that the output is readable.

In Lebab's case you'll want the output to be 100% the same as input when no transforms were done - so Lebab would not mess up whatever strange coding convention one has, and you could look at the diff of code before/after and only see the actual transformations without any whitespace noise.

So it seems to me that this is not a feasible approach until Babel adopts a code generator more similar to Recast.

@nene
Collaborator
nene commented Jun 12, 2016

On further investigation I discovered that it might be feasible to use the parser of Babel - Babylon. The AST it generates is mostly compatible with ESTree format used by Esprima and others, but there are some minor differences.

@benjamn
benjamn commented Jun 13, 2016

Recast is parser-agnostic; you can call recast.parse(source, { parser: whatever }), as long as whatever is an object with a .parse method that produces AST nodes with .loc information.

@nene
Collaborator
nene commented Jun 13, 2016

@benjamn Yep, Lebab is already using that run Espree parser instead. The problem with Babylon is that it doesn't produce AST that's fully compatible with ESTree spec.

@hzoo
hzoo commented Jun 18, 2016 edited

@benjamn So we could switch out babel-generator for recast.print to keep original formatting (assuming you wanted something like jscodeshift/eslint autofix/run babel like babel src --out-dir src?

@hzoo
hzoo commented Jun 29, 2016 edited

A babel transform/babel = babylon + babel-traverse + babel-generator so you could modify the ast with a babel transform and output to something that can consume a babel AST (recast)?

@benjamn
benjamn commented Jun 29, 2016

Well, Babel is optimized for code generation performance, and preserving formatting takes a bit more effort, but recast.print will work with any AST that recast.parse parsed (as long as the nodes have .loc.{start,end} information).

@hzoo
hzoo commented Jun 29, 2016 edited

Ok I'm just wondering what we can do to make recast + babel interop better if that's possible. I think it would be nice to be able to write a babel transform like babel src -d src --presets=lebab --generator=recast and it uses recast instead of babel-generator to preserve source (codemod, autofix, etc). I've experimented with simple transforms like http://astexplorer.net/#/sskXYR4icD/2

@benjamn
benjamn commented Jun 29, 2016

If this works (and I don't know why it wouldn't), then it shouldn't be hard:

var recast = require("recast");
var ast = recast.parse(source, { parser: require("babylon") });
var result = require("babel-core").transformFromAst(ast, {
  presets: ["lebab"],
  ast: true
});
console.log(recast.print(result.ast).code);
@hzoo
hzoo commented Jun 30, 2016

@benjamn would recast have to parse it like recast.parse(source, { parser: require("babylon") }); or can you still use babylon on its own? Just wanted to know what changes would need to be made to babel to support this for a regular babel setup (sounds like we would need an option to use a different parser/generator in babelrc)?

@benjamn
benjamn commented Jun 30, 2016 edited

Letting recast.parse do the parsing is not essential, in principle, but it does have several advantages:

  • The recast.parse code can pass the options it needs to the parser, reducing manual configuration.
  • The .loc.{start,end} information in the parsed AST is guaranteed to correspond to the original source, since no other code has had the chance to modify the AST. In some cases recast.parse even fixes bugs related to .loc information for the various Esprima-like parsers that it allows.
  • The real magic of recast.parse is that it makes a copy of the AST, and links every node in the copy to its original node via an .original property. This shadow-AST is what enables recast.print to perform tree diffing and reuse original source code when possible.

In other words, you could parse the AST yourself, and then call some other recast.* API to make the copy and create the .original linkage, but recast.parse is a simpler interface for that, I think.

@hzoo
hzoo commented Jun 30, 2016 edited

Ok I can test it my modifying babel's transformation step to take in a different parser/generator.

I think the last thing is to accept babel AST nodes in ast-types? Since Error: unknown: {type: ObjectProperty} does not match type Printable Or just modify them like before? (Looked at babel/babel@eac9e9b) seems to work as a experiment

I recall benjamn/ast-types#132 too - should we make separate types for babel's 6 ast differences?

@hzoo
hzoo commented Jun 30, 2016 edited

@benjamn yay something like this seems to work babel/babel#3561

or it would be cool if we could subsitute with babel-types

@mohebifar @nene I can help with converting when it's possible. Lebab can be a babel preset with plugins (could be a monorepo with lerna as well)

@hzoo
hzoo commented Jul 15, 2016 edited

As a update for this to happen:

  • babel/babel#3561 in babel-core needs to land in some form (probably a option to pass a string (similar to a plugin) that resolves to a module with a parse method and a generator method) - I'm testing in my local astexplorer
  • benjamn/ast-types#162 update ast-types with babel 6 types - (need to fix some tests)
  • benjamn/recast#299 update recast printer with the same types - (need to fix some tests)
@hzoo
hzoo commented Sep 29, 2016

babel/babel#3561 has landed! So writing a plugin that doesn't include the different AST nodes should work now https://github.com/babel/babylon#output

@hzoo
hzoo commented Nov 4, 2016

Merged the 2 prs ^ so this is actually a possibility now! example here: fkling/astexplorer#161

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment