Parser plugins

[natew]

I'm enjoying playing with Babel plugins so far and have been looking at sweet.js as well. Unfortunately using the two together (as I've seen talked in threads here before) isn't great.

How could one implement a similar macro system for JS but using Babel/Acorn rather than Sweet/Esprima? Are there plans for custom syntax support, I would imaging this issue really falls more on the AST generation side, but then I realize Babel already has custom syntax like decorators and JSX.

I guess I'm a little green on how it all works, but curious on plans for this or ways to implement this today. As a minimal example I'd like to do some of the following transformations:

#x = 1

to:

State.set('x', 1)

And also

component XYZ {
  <sortaJSX></sortaJSX>
}

into:

class XYZ extends Some.Component {
  render() {
    return { name: 'sortaJSX' };
  }
}

Just throwing out a couple examples of things that would be really awesome and enabled by macro support. For now I'm hacking together stuff with webpack loaders with regex replace, checking into PEG, and also reading through Babel, but would like to see if there's any interest in this, and would appreciate pointers in the right direction.

[sebmck]

Why do you want macros?

[natew]

To make a DSL for things I'm currently writing out at length. Just to play with adding features to the language. I want to be able to write something that transforms my code like shown in those two examples.

[natew]

Wouldn't macros make your life easier? In that really all a macro is is what you're using to make Babel in the first place, so to open that power up to people would allow anyone with a new spec the ability to implement it themselves, let people try it, and eventually have it submitted back to the core.

I find it very exciting. But from your reply I'm guessing you are against them in some sense. Care to explain? I'm sure there are many angles to this, so I'm curious what your thoughts are.

[sebmck]

How could one implement a similar macro system for JS but using Babel/Acorn rather than Sweet/Esprima?

There's no way at the moment.

I guess I'm a little green on how it all works, but curious on plans for this or ways to implement this today.

There are plans to add something like this, although a macro interface isn't the direction it'll be in. More like just some way to hook into the parser and code gen logic.

Wouldn't macros make your life easier?

Not really. I don't think throwing syntax at something is the solution.

In that really all a macro is is what you're using to make Babel in the first place, so to open that power up to people would allow anyone with a new spec the ability to implement it themselves, let people try it, and eventually have it submitted back to the core.

Are you referring to simply allowing pluggable syntax into the parser? That's different to macros.

I find it very exciting. But from your reply I'm guessing you are against them in some sense. Care to explain? I'm sure there are many angles to this, so I'm curious what your thoughts are.

I think macros are misguided. You're trying to reduce repetition but add an extremely large and convuluted layer of abstraction on top instead. The two examples you provided are pretty minimal. I find that 99% of macros are basically find-and-replace like things where it's a 1:1 mapping, not really powerful or necessary IMO.

Whenever you try and do anything complicated you get a massive clusterfuck of syntax productions which makes things infinitely more complicated than dealing with an AST. Take rest parameters for example, you have to support all the following forms:

function foo(..rest) {}
var foo = function (...rest) {};
var foo = (...rest) => i * i;
var foo = (...rest) => { return i * i };
var foo = { foo(...rest) {} };
class Foo { foo(...rest) {} }

I'm unsure what you're asking for specifically? Just way to define custom syntax or an actual macro interface? Because they're both very different things.

[btholt]

@natew I have a couple of suggestions:

• Try ES6 Macros. This will allow you to bypass dealing with Babel and just deal with Sweet.
• Wait for Babel to settle a bit more. @sebmck has stated that a plugin system is likely in the future.
• If you're feeling adventurous, I imagine it may be possible to rip the parser out of one, replace it with another parser, and then write a glue layer between the two, and then resolve the mess that you'll be left with. That's the only way I can see to actually do what you're asking for.
• Perhaps you could even just write a simple macro layer on top of Acorn. That would seem much simpler than trying to port Sweet/Babel.

All of that said, I'm inclined to agree with @sebmck. While at reddit, we toyed around using some Sweet.js macros with React after Sweet.js got readtables. We wrote a little app in it but we found it to be an absolute rat's nest to debug; not only are you debugging your code but you're debugging your tools too; this added level of complexity made the proposition of going forward with Sweet.js a total nonstarter. We promptly rewrote the little experiment we did in plain JSX.

[natew]

@sebmck I think I didn't clarify really, for me macros vs plugging into parser is all the same, I just was wondering about the ability to do the transforms (result rather than method). I appreciate your reply. In an ideal world I'd be able to design a new language myself, and not have to worry about optimization and all the other things that come with that. But for now there are certainly a few things that I could do with JS today by extending syntax that would vastly simplify what I'm writing and even simplify things in many ways. In fact, I'd want to combine these syntax extensions with restrictions (think: soundscript, remove the bad parts, but today, with Babel plugins).

Either way, it's great to hear that custom parser plugins are on the table. Count this as my 👍 for them.

@btholt Awesome recommendations. I hesitate to go with sweet.js/esprima, given the amazing work @sebmck is doing and support generally around Babel. Though sweet.js pretty much solves my problems today, so perhaps it's the practical solution to go with for now. I hasn't seen the es6 extensions. Your other suggestion of writing a macro layer on Acorn to me would be the next best option.

What I think is this: if you create macros not to ease common tasks but with the intent of drastically simplifying the interface a programmer needs to use (ie: restricting the surface area of JS while moving very few new ideas out there into more declarative interfaces, which could be achieved with a combination of macros and babel plugins), the double edged sword of macros could be balanced well.

To me, React is getting closer to what we need. But there's a lot of boilerplate still. Especially for beginners. JS will move from a scripting language, to an interface for building UI's, and it would be cool to even prototype a system where you can think that way and avoid writing so much boilerplate. Even if it's brittle/hacky, it would help show new ideas and eventually push JS forward much in the way CoffeeScript did.

Anyway, I'm rambling. Appreciate both responses. Let me know what would be the most productive way I can help move things forward.

[btholt]

@natew: Totally on board with what you said. JS is definitely permeating into every device and venue. It's also frustrating to have to remember and deal with a lot of boilerplate. The way I've chosen to deal with it are twofold:

• Sublime snippets. Especially helpful with React. It's nice to just say cwm-Tab and have it autocomplete out to a componentWillMount method. While the boilerplate code is still there, you're not stuck typing it every time.
• A living and enforced code style guide. While something like macros could make it so you couldn't use some parts of the language, a style guide just makes it so you don't. Couple your style guide to the pluggable babel-eslint and git hooks, and you accomplish the same thing: a subset of your choosing of JavaScript that's enforced.

I know this may not be what you're looking for but I wanted to share that I have the same problems/feelings and I feel like those two things helped me solve it for myself satisfactorily. In any case, I'm interested to hear how you solve it and hope it works out. 😄

[monsanto]

@natew Is there anything actionable to do here, or can we close this topic & resume discussion when Babel has better support for your use case?

[natew]

However you run the repo is fine with me, I think I've got a better feel for things. I tweeted to the ticket here to for anyone else to chime in.

[sebmck]

I'll keep this open as a tracking issue for parser support in Babel.

[RnbWd]

Interesting talk from the creator of CoffeeScript, he mentions babel and uses source code in some slides.

@sebmck

I think macros are misguided. You're trying to reduce repetition but add an extremely large and convuluted layer of abstraction on top instead. The two examples you provided are pretty minimal. I find that 99% of macros are basically find-and-replace like things where it's a 1:1 mapping, not really powerful or necessary IMO.

Transpilers have so much potential for transforming how we interact with the web stack (which I've always ironically referred to as the 'tower of babel'). TC39's is the 'arbiter of truth' because we had no choice but to use what the browsers provide. JSX is THE prime example of something that would have never came into existence from a committee. IMO, it's the tip of the iceberg. Babel has so much influence at the moment, and plugins are like the key into its library of infinite possibility.

Anyways... I'm interesting in somehow integrating postcss within the plugin structure so that we can interpolate css into js-styles. But I'm not looking for an answer to that question here. I just wanted to say plugins are really powerful, macros / transpilers are very similar, and theoretically any semantic structure within the language can be turned into another, and I think we should encourage that kind of thinking instead of only looking towards standards

[m1sta]

I agree with @RnbWd that JSX is a great example of the value of macros. I find it difficult not to include all of the babel plugins and transformers under the label of 'macro'. Making it easy for people to prototype and share potential future javascript syntax improvements seems like part of the core goal of babel.

Acorn in theory supports plugins already, although the author has been pretty open about not intending to spend any time improving plugin support in the forseeable future. @natew the espree team might be worth keeping an eye on too, looking at the code they seem to be slowly progressing towards better parser modularity.

Whatever happens it'll be important to learn from sweet.js experience. It's still a great project but I can't imagine anyone wishing that babel was written as sweet macros - and in the end the goal should be, IMHO, that "everything that babel does should probably be built as a macro" with babel being a framework for running a pipeline of macros to generate an ast, transforming that ast, and generate code for a js target.

[sebmck]

Babel is as much a set of macros as any compiler is. Would you label GCC or CoffeeScript as a set of macros?

[m1sta]

@sebmck Fair point! Perhaps, I'm not well informed enough to argue to semantics. My gut feel is that one factor here is that babel allows standard ES5 to be written, with extensions, but with the result being ES5. It's not an entirely new language from scratch. I'm sure any macro tool could probably be easily butchered to cross whatever threshold lies between 'macro framework' and 'compiler'.

[natew]

When we consider stuff like JSX, flow types, and extremely early stage proposals that are implemented in babel, it's more than just a 'future js' compiler already.

Anyway babel is doing great. I think the next big thing in web development will be someone who takes JS and removes some of its warts, and people adding in macros for stuff we are discovering as useful, like flow and JSX. It's just strange to only support one companies' syntaxes. There is so much untapped power that people would bring with a really great plugin system on the language level. It would be an amazing boost to the web in the long term to unlock all those creative minds.

[sebmck]

I'm not against adding this type of functionality. I'm just against people suggesting there be some massive rush to hastily implement it instead of thinking it through.

[RnbWd]

IMO what separates babel from other compilers like GCC, Coffeescript, etc. is its modularity, openness, and simplicity. The difference between a 'macro' and 'compiler' becomes less distinguished if the compiler is written in the same language and is easily modified. Many people have taken the source code of coffeescript and rewrote it to compile their own custom 'syntax'. Babel's architecture could enable an entire ecosystem of syntax transforms.

@sebmck I agree this needs to thought through carefully and not rushed or hastily implemented. But is not already implemented? I know that badly written plugins could wreck havoc, but it doesn't have to be a concern of babel-core.

[sebmck]

@RnbWd Custom syntax hasn't already been implemented, no.

[RnbWd]

@sebmck http://babeljs.io/docs/usage/plugins/manipulation/ ?

[sebmck]

@RnbWd That doesn't say anything about custom syntax?

[RnbWd]

@sebmck if this isn't considered custom syntax I don't know what syntax is. Then again, I've been reading Finnigans Wake for the last 2 days so...

"Let us leave theories there and return to here's here. Now hear. 'Tis gode again. The teak coffin, Pughglasspanelfitted, feets to the east, was to turn in later, and pitly patly near the porpus, materially effecting the cause." - p.76

edit: meant to also reference this: http://babeljs.io/docs/usage/plugins/scope/#rename-a-binding-and-it-s-references

[sebmck]

That's about manipulating the AST. There are three different parts of Babel:

Plugins let you hook into the second one. Syntax needs to be added into the parser.

[RnbWd]

@sebmck I understand now :)

That actually makes sense.

[roman01la]

Hello. I believe macros and vars tracking would enable to build a plugin for Immutable Data Structures proposal to replace the syntax with mori or Immutable. At least this is something I'm looking into.

this will be translated

const a = #[1, 2];
const b = #[1, 2];

console.log(a === b);

into this

const a = mori.vector([1, 2]);
const b = mori.vector([1, 2]);

console.log(mori.equals(a, b));

[mgcrea]

Just stumbled on this while trying to properly parse angular2 parameter annotations eg. @Optional router: Router, this is an example of a parser "hack" that used to work on former babel versions. For now I haven't found a way to achieve the same thing with the latest babel version.

So a big 👍 for pluggable syntax into the parser.

[dead-claudia]

Could the parser accept plugins here and here and the tokenizer here? Those appear to be the most extensible places.

As for semver, I think only the basic pp.parseExpression, pp.parseIdent, etc. should be part of the blessed API, and should only be semver-minor when breaking (as most changes would be), since it won't affect the rest of the platform, not even the transformer plugins.

[sebmck]

Introducing exotic node types will require a change in global state which is gross and impure.

[dead-claudia]

@sebmck Good point.

[lydell]

Perhaps sweet.js could be used for custom syntax.

[dead-claudia]

That's probably better than what any parser plugin could do. Although I
feel Sweet.js syntax is rather verbose.

On Tue, Sep 29, 2015, 01:13 Simon Lydell notifications@github.com wrote:

Perhaps sweet.js https://github.com/mozilla/sweet.js/ could be used for
custom syntax.

—
Reply to this email directly or view it on GitHub
#1351 (comment).

[babel-bot]

Comment originally made by @thejameskyle on 2015-11-20T19:31:18.000Z

Marking as low priority for now.

---

Comment originally made by @STRML on 2015-12-02T05:03:18.000Z

Would be nice to have this support without hacking the parser and hoping for a PR to be accepted.

I was looking to implement the syntax described here for JSX implicit do expressions: reactjs/react-future#35 (comment)

Unfortunately it's not possible because the parser doesn't get far enough to allow a plugin to wrap JSXExpressionContainer contents in a DoExpression.

---

Comment originally made by Andrew Depue (andrewd) on 2016-01-21T18:56:05.000Z

My vote would be to expose a parser/syntax plugin API and just let people know that their plugins may break in future releases. As a user of Babel I wouldn't mind this. We lock our dependencies down for our application anyway, and we don't upgrade dependencies until they have been tested with our app. If we are using a syntax plugin and it hasn't been updated by then - well, we can fix it ourself and submit a pull request to the plugin's project.
In reality, though, we will be using our own syntax plugin and have no problem refactoring it when we move to new Babylon releases. It's easier than maintaining a fork of Babylon.

---

Comment originally made by @lukehorvat on 2016-04-24T11:08:05.000Z

Given that it may be quite some time until this is ever resolved, I've built a tiny wrapper around Babel v6 that supports "advanced" plugins, aptly named Babby.

https://github.com/lukehorvat/babby

"Advanced" plugins can modify Babel's parser to add new tokenizer token types and AST node types.

Anyone who is interested, feel free to check out Babby and help me make it better.

[hzoo]

There's a lot of work in Babel to be done already so I don't think we want to take this maintenance burden any further. If you need to experiment I would suggest forking babylon for now (you can pass this into babel via parserOpts

{
  parserOpts: {
    parser: "babylon-fork-npm-pkg-here"
  }
}

or sweetjs, etc.

Also closing since at this point babylon is in a separate repo there's no immediate plans for this https://github.com/babel/babylon.