Proposal: Plugin Architecture

[GeoffreyBooth]

I’m thinking that maybe the way to handle things like JSX (see #4529) or type annotations is to build in hooks for plugins into CoffeeScript. Specifically, there would be two: preprocessors and postprocessors. We already have one preprocessor already: invertLiterate, the function that separates the comments from the code for Literate CoffeeScript. That function basically just parses each line and adds # before any line that isn’t indented. What if we split that function out into its own module?

When the compiler runs, it would see that the “literate” module is loaded and that it provides a function to run in the preprocessors hook, and so CoffeeScript would run that function before parsing any code, the same way it runs invertLiterate now. The function would take the code string as input and return a modified version of the string as output (along with source maps, if necessary). And likewise for functions provided for postprocessing. Someone could write a plugin that provides a postprocessing hook for transpiling through Babel, for example. Some plugins could provide both types of hooks, for example to remove code before passing it through the compiler and restoring it afterward. Plugins could also specify dependencies, like that they should always run before or after a certain other plugin, so that people could enable multiple plugins to have several functions run in sequence in the same hook.

This opens the floodgates for any extensions to CoffeeScript that people might want to enable on a project-by-project basis, without having to fight to get the modification included into the language itself. This could be the solution for the “Coffee Tags” proposal for adding a JSX-like syntax; or for macros.

The problem to solve would be how plugins could pass non-CoffeeScript code through the compiler. Backticks are currently allowed only where the embedded JS can be an expression; a line like return `<h1>` @title `</h1>` is disallowed. Block comments are even more restricted, allowed only as statements or inside object literal assignments; something like ### String ### name = 'Joe' is also disallowed. Is there a way we could expand the types of places that embedded JavaScript or block comments could be permitted? I understand that such tokens can’t simply be allowed everywhere, since lines like eat food for food in foods when `/* String */` food isnt 'chocolate' cause significant new code generation and therefore it’s ambiguous where the embedded JS /* String */ should go in the resulting output. But perhaps we can increase such tokens’ utility enough to cover most cases that people would want to use them for, for example for JSX tags or type annotations. Such improvements would fix #4464.

An even more ambitious plugin architecture would involve hooks for new types of tokens in the lexer itself, like what the “Coffee Tags” proposal adds. This would spare plugins from having to implement their own lexing and parsing logic. But for the first version of this, I think some way to pass through embedded JavaScript or /* */-style comments would solve several of the most common use cases.

[lydell]

would solve several of the most common use cases.

What are the most common use cases?

[mrmowgli]

Off the top of my head some of the ones I have seen are: Linting ala Coffeelint etc Translitterating ala Babel plugins Hybrid Syntax Trees ala JSX Packaging ala Webkit And ??? There are quite a few that fit as AST transforms, including the Babel plugins and Coffeelint. I could see other langs like Pug being usable that way as well.

[GeoffreyBooth]

What are the most common use cases?

The ones I had in mind were:

• Literate CoffeeScript itself (we’ve discussed how it doesn’t really belong in the core compiler)
• JSX or other embedded templates, or embedded any-other-language
• Type annotations, if someone wants to integrate with Flow or create an unholy hybrid of CoffeeScript and TypeScript
• Transpilation down to ES5 without needing a separate build step

@mrmowgli names two other good ones. Basically this would be used for things that don’t belong in the language, but if people want to use them then some level of integration with the compiler is necessary.

We might even see some creativity like what’s happened in the Babel community, where people create plugins for not-ready-yet ES features, or not-even-JS features like JSX.

[lydell]

Random note: Parser plugins is not a simple problem. https://github.com/babel/babylon#faq

[vendethiel]

Yeah, everything in this screams complexity. Our lexer/rewriter is a monstruous beast. We'd have to take a look at sweetjs, readtables/macros in languages...

[GeoffreyBooth]

Then maybe we never give people access to the parser or lexer or nodes. A lot can be accomplished by just allowing a pass over the code string before the compiler sees it—that’s exactly what Literate CoffeeScript does.

My JSX proposal, or at least the “half-assed version” where anything inside of XML tags is JS, could be implemented with such a minimal plugin system. If we improve the handling of embedded JS or block comment nodes, so that I could create a plugin that wraps each XML tag in backticks and the compiler just parses around them, then I could create a full-blown JSX plugin. Even still, that’s just improving the parsing of embedded JS, which we should probably do anyway—such improvements still don’t provide access to the lexer or nodes etc.

[mrmowgli]

This plugin issue has been going on for years, and it needs to happen at some point. There are too many people pushing for a core change to the language, and letting people ‘kind of’ use a feature is going to lead to a million issues where ‘it should do xxxx because the new version of yyyy allows it’. I also suspect this is something that doesn’t need to push back releasing CS2.

[lydell]

So you're saying that a plugin takes a string of code as input, parses it, compiles it to a string of CoffeeScript and then passes that to the CoffeeScript compiler? Yes, that's how literate mode works – and it works since there's no nesting. There's no markdown inside the code, just code at specific toplevel delimiter-free positions inside the markdown. I can't see how JSX or type annotations could be done without parsing. For example, you wouldn't want to touch JSX-looking stuff inside strings or regexes. To me, the whole system sounds like giant hacks.

I just noticed that Acorn supports plugins, such as the acorn-jsx plugin. Perhaps we could take inspiration from that.

[jashkenas]

Literate CoffeeScript is simple and hacky, and we shouldn't be adding hooks for more hacky preprocessing like it.

You can already preprocess your code before you pass it to CoffeeScript in the first place — however you like.