[pug-code-gen] Rewrite using babel

[jeromew]

From @ForbesLindesay on November 6, 2015 10:1

babel does a really good job of generating source maps. Generating ASTs will also help us only generate proper, syntactically valid javascript. As a bonus we can rapidly run any babel plugins we want on the output for optimisation or implementation of advanced features.

One of the restrictions this will impose is that unbuffered code blocks (e.g. - var x = 10;) will now need to be valid statements. This forces people to use our built in constructs for if, else, each etc. I don't think that's a bad thing though, and we can make up for it with much better error messages for syntax errors.

I think this is too big a feature to block the 2.0.0 release, which I really want to roll out this weekend. I do think we should hold off on plugins for code-gen until this is complete though.

Copied from original issue: pugjs/pug-code-gen#21

[jeromew]

for info I am keen on trying to see how then-jade and react-jade can be built on the new 2.0.0 toolkit ; these 2 libraries modify the code-gen.

by 'Rewrite using babel' you mean that jade-code-gen should feed AST directly into babel ?

[jeromew]

From @ForbesLindesay on November 6, 2015 10:39

Yes, that's what I mean by re-write using babel. In the short term, then-jade should fully fork the code-gen module, but will benefit from a more stable and defined AST structure. In the long term I hope we will have a plugin architecture that allows overriding just the mixin parts of code-gen. react-jade is fairly irrelevant here because it's such a total fork of code-gen anyway (it doesn't share any code-gen code with jade core).

[jeromew]

I'll look into forking this for then-jade once 2.0 is publish ; I think the buf += path was chosen over the buf.push so the diff will be a bit bigger than before.

regading react-jade I agree that it has a lot of specific React tricks. The jade transformed into a call tree as something generic to it though ; a bit like http://mochi.github.io/mochikit/doc/html/MochiKit/DOM.html was doing

[jeromew]

From @ForbesLindesay on November 6, 2015 11:56

I suppose all you would need from a plugin API is the ability to override buffer, bufferExpression, visitMixin and visitMixinBlock (you no longer need visitEach because we no longer generate an immediately invoked function for each blocks). We could create a simple plugin system that let you do that, but I think it would end up being brittle and would need to be replaced when we moved to babel anyway. A better plan (I think) would be to continue with your current inheritance model, which is actually pretty simple and effective.

[jeromew]

I'll look into that ; do you have a sample repository that shows how to derive a new compiler from jade 2.0 ? currently, the inheritance in then-jade is straightforward but I am not sure how to fork only jade-code-gen.

would you copy/paste https://github.com/jadejs/jade as a new project and start from there replacing jade-code-gen with the forked version ?

as a side question, I guess i would be interested in trying to rewrite jade-code-gen using babel as you suggested. Regarding coding conventions, have you seen a sane example project where AST is direcly built and fed into babel ?

[jeromew]

From @ForbesLindesay on November 6, 2015 13:32

You can create a new CodeGenerator class by inheriting from require('jade-code-gen').CodeGenerator. For now, I would copy paste https://github.com/jadejs/jade as you suggest, although we may come up with a better solution down the road.

No, unfortunately I'm not aware of any projects that do this already, but I've asked Sebastian McKenzie what he thinks and he felt it was a pretty sensible approach. I'll ask if he knows of any projects currently doing this/has any advice on how to get started.

[jeromew]

From @TimothyGu on November 13, 2015 2:36

I'll ask if he knows of any projects currently doing this/has any advice on how to get started.

Have you done so yet? Frankly, the notion of generating a fully qualified JavaScript AST from {} sounds pretty difficult, so an example would really help.

I also wonder how one can generate source maps with a different language, but I guess adding location information to the Babel AST should suffice.

[jeromew]

I looked a bit at the babel source code. Even if the code of jade-code-gen is not very big, I suspect that generating from {} will need some helper functions.

The basic node construction feature of the AST seem to be given by https://github.com/babel/babel/blob/master/packages/babylon/src/parser/node.js

It might be feasible with https://github.com/benjamn/ast-types but I did not check how compatible the ASTs are.

[jeromew]

From @ForbesLindesay on November 17, 2015 16:24

He wasn't aware of anything, but gave me a few pointers on where to get started. We should use:

• babel-types for building the AST (I have a pull request to generate some basic documentation at Generate documentation for babel-types babel/babel#2933, for now, you can read the docs in ForbesLindesay/babel) (we will also need to extend/wrap this to add location info to the nodes, but that shouldn't be very difficult).
• babel-traverse for running any optimisations/plugins on the generated AST. We can use this to run generic babel-plugins (for ES2015+ features an optimisations) as well as run custom plugins for doing things like inlining mixin calls and combining multiple separate calls to add strings onto the buffer. This will allow our own code-gen phase to be much more naive and then leave the work of building optimised code to babel.
• babel-generator to generate the JavaScript and source-map (We will need to fork/extend this to support source maps that originate from multiple files, but that's just a case of replacing babel-generator/src/source-map.js with something that takes the filename from each individual node)
• babylon for parsing user supplied JS expressions and statements.

[jeromew]

I have vaguely started to look at this (very low priority on my side).

While trying to see what I could do with the buffer() method, I wonder whether every new buffer addition should lead to a pug_html = pug_html + ".." or if we should keep the "smart concatenation" code that is inside buffer()

clearly, a babel plugin could take care of such concatenation afterwards but I wondered what you think because every babel plugin that we will add will have its own running cost.

[jeromew]

From @ForbesLindesay on September 12, 2016 15:3

I think we need to at least make sure that consecutive strings result in pug_html = pug_html + 'SomeString'; not pug_html = pug_html + 'Some'; pug_html = pug_html + 'String';. I'm not sure if we should bother reducing expressions, since it's definitely detrimental to performance after a certain point (I don't know what the optimum number of concatenations per statement is).

I don't have a strong opinion on whether this should be done as part of generating the initial code or in a babel plugin. I think it depends a lot on how difficult the plugin would be to write.

[jeromew]

I worked a bit on this again. I convinced myself that at this stage, it is better (if possible) to try and generate the closest javascript to the current generated javascript (it seems easier to me to compare the 2 generated javascript in parallel). It should be easy to refactor once we have test-parity.

Now I hit a problem because of how js strings are generated. In pug we use js-stringify to work out some escaping before inlining the strings in the output javascript.

With babel, declaring literals with t.stringLiteral("<p>"), we get strings with

"<p>"

instead of strings like those pug currently generates
"\u003Cp\u003E"

so we lose the escaping, js-stringify seems to not help with babel because t.stringLiteral("\u003Cp\u003E") produces code with double escaping of slashes.

Can you tell me more about why we do this escaping and if when generating the whole code from AST we would still need it ?

I currently don't see a way to have babel generator generate u-escaped brackets from literals the way js-stringify does; I would appreciate some feedback on this.

The closest issue I found is babel/babel#4478

note: maybe this has to do with incompatibilities when the generated js is written inside an html script tags

found on the web: HTMLEscape appends to dst the JSON-encoded src with <, >, &, U+2028 and U+2029 characters inside string literals changed to \u003c, \u003e, \u0026, \u2028, \u2029 so that the JSON will be safe to embed inside HTML script tags. For historical reasons, web browsers don't honor standard HTML escaping within script tags, so an alternative JSON encoding must be used

Extract found here

by the way it is weird that in this extract that choose \u003c, \u003e, \u0026, \u2028, \u2029 while js-stringify chooses \u003c, \u003e, \u002F, \u2028, \u2029.

it could maybe be a babel plugin used when generating literals (?)

[jeromew]

From @ForbesLindesay on October 27, 2016 9:23

This is an interesting point, that I had not considered. The reason we generate those characters, instead of < etc. is to ensure that it is always safe to embed a compiled pug template directly inside a <script> tag. In reality, the only sequence that needs to be escaped is </ (according to the spec). I think in actual browsers, (someone should check the latest spec) you only need to escape the sequence </script> so maybe we could check for that and either:

1. fork babel-code-gen to escape the characters in </script> when it outputs a string literal
2. or generate a concatenation so we output '<' + '/script>'

Option 1 should produce much more efficient code. This may be something babel is keen to offer, but I don't know.

I think long term we should avoid escaping everything like js-stringify does. I took the approach of escaping everything in js-stringify to be as safe as possible since I (and many other people) was using it to embed user generated info in inline scripts in JavaScript, i.e.

script.
  var CSRF_TOKEN = !{stringify(csrfToken)};

The overhead of this extra escaping probably does slow pug down and certainly makes for bigger/less readable output. We should try and minimise this overhead.

[jeromew]

ok. At this stage I think that I will not take this escaping into account. Since t.stringLiteral evaluates the string, there is no way I think to avoid the internal un-escaping. So it will have to be handled at code-generation (either by a fork or a PR on babel-generator)

babel generator uses https://github.com/mathiasbynens/jsesc for literal escaping and there is a isScriptContext option :

The isScriptContext option takes a boolean value (true or false), and defaults to false (disabled). When enabled, occurrences of </script and </style in the output are escaped as </script and </style, and <!-- is escaped as \x3C!-- (or \u003C!-- when the json option is enabled). This setting is useful when jsesc’s output ends up as part of a script or style element in an HTML document.

currently babel-generator does not allow to specify jsesc options - https://github.com/babel/babel/blob/master/packages/babel-generator/src/generators/types.js#L132

[jeromew]

for information i could force the escaping using an undocumented feature of stringLiterals via

 var lit = t.stringLiteral(this.lastBuffered);
 lit.extra = { rawValue: lit.value, raw: stringify(this.lastBuffered) };

not sure if we want to keep this is what babylon does when it parses escaped strings.

[jeromew]

From @ForbesLindesay on October 27, 2016 13:44

That's a neat option. We should probably update js-stringify to just do what jsesc does providing that is properly secure when faced with user generated content. I'm tempted to say lets use that hack for now so we don't forget to resolve this later. Up to you though.

[jeromew]

for info i hit into other difficulties with statements like

- if (true)
  p something
- else if (true)
  p something
- else
  p something

where code blocks have to be babylon'ed while not parsable in their own right. else if (true) is not valid javascript without an if ; else either. So I am considering detecting these constructs to be able to map them to their ifStatement(consequent, alternate) equivalents. But it may lead to incompatibilities with complex code blocks, like - var foo="bar"; if (true)

the freeform javascript + 2 options we currently have

- if (true)
- if (true) {
- }

make it not so easy handle all cases.

[jeromew]

I did a custom implementation for if/else if/else code blocks but there are more interesting things with

  - items.forEach(function(item){
    li= item
  - })

in order for babylon to parse cases like these (or more complex cases) , we need a strategy. Maybe balance all opened '(' and '{' to make the thing parsable, with a fake statement inside like t.identifier("placeholder") that we can use to find the block statement where items should be inserted.

but I don't know what level of javascript hacks inside the templates we want to support + there could be hairier things like maybe

  - try {myFunc(
    li a
  - ,
    li b
  -)} catch(e) {
    li c
  - }

whatever sense that may have (not tested).

If you have ideas I am interested !

[jeromew]

@ForbesLindesay the "freeform" js that some people may currently inject into pug templates is I think going to be difficult to keep as a promise.

Currently, to handle

- items.forEach(function(item){
  li= item
- })

To keep the current visit flow, I need, on the items.forEach(function(item){ visitCode to detect that there are unbalanced "})". Knowing this, I do some sort of babel-template on items.forEach(function(item){BLOCK}) so that I can inject the code.block ast into BLOCK. Then when I detect code.val == '})' I discard it.

babel-template currently does not handle blockStatement, only expressions so it cannot be used as is to do the injection. A solution would have to be found to do this, or at least find the BLOCK marker.

of course, detection }) and discarding does not seem bullet proof because there are probably many cases I do not even think of like

- items.forEach(function(item){
  li=item
- ;{
  li=item
- }})

There could be another option, like buffering all code tokens (while addind markers) until we have a buffer that is balylon parsable (does not throw). While doing this, we would capture the AST of the blocks in their corresponding marker variable.

In the second example, that would be something like items.forEach(function(item){ BLOCK1 ;{ BLOCK2 }})

That would probably be more bulletproof but do you think that detecting a "parsable" block is a correct end-marker for the buffering algorithm ?

Clearly I fear that we may create hard to fix backward incompatibilies for some people who have created monster-templates for their own good reasons.

I'll do a pause on this now. Right now my code is not in a state of commit I would need to clean up non-generic code on this balancing issue. 212 tests passing, 29 errors (including these balancing problems + mixins that I have not started)

[jeromew]

I did a bit more work on this (hard to pause ;-)

The second algorithm kind of works and is the best I can come up with at this stage. I had misunderstood something in babel-template and we can in fact use it without a modification.

now the problem I have is that in

- if (true) {
  li something
- } else {
  li else
- }

the algorithm works well, since the first parsable snippet is if (true) { .. } else { .. }

but

- if (true)
  li something
- else {
  li else
- }

breaks because if (true) .. is parsable and self sufficient. else { .. } is lost because non-parsable. So "parsable" is not a good signal to know the end of stream of code blocks with dependency. I may have to keep a flag to flush a stream of code blocks once we hit a non-block token at the same indent level or eof.

[jeromew]

From @TimothyGu on October 28, 2016 19:53

I'm not sure how applicable this is, but can you detect if there is a block attached to a specific Code, and if so automatically add the brace?

[jeromew]

OK. I managed to rewrite the whole thing and have all 241 pug tests pass :-)

I realize that in your first message @ForbesLindesay you said

One of the restrictions this will impose is that unbuffered code blocks (e.g. - var x = 10;) will now need to be valid statements. This forces people to use our built in constructs for if, else, each etc. I don't think that's a bad thing though, and we can make up for it with much better error messages for syntax errors.

The current algorithm I choose to avoid this is to buffer all the code blocks of the current level (I modified visit to always pass the parent so visitCode can know where it is wrt its parent.

the current impl is on https://github.com/jeromew/pug-code-gen, on the babel branch.

I'd appreciate @ForbesLindesay and @TimothyGu If you have some time to take a look at this and tell me what you think. At this stage, I mainly tried to make the tests pass while keeping the code acceptable, but this could use some improvements.

Things that are left to do :

• clarify what we want for escaping in this.buffer

• see how we can forward the line/pos on the AST tokens to generate sourcemaps

• decide if we want to keep the "freeform" javascript or restrict code blocks to parsable blocks

• remove with and replace it with a custom implementation using babel-traverse to avoid reparsing the whole js

• see if we can improve speed

• refactor code to abstract the creation of functionExpressions and pug_html concatenation

• improve readability of visitEach

• find bugs in the implementation that are not covered by pug tests

• create error message when a serie of code blocks is not parsable (catch is already implemented)

• maybe earlier parsing of attributes I did not check how it is done now

  I thinks it is important that we discuss the pros/cons of this before going forward to check if this is something we want to push forward.

The code is less error prone (no dangling brackets) and more clear once you get used to reading ast meta code.

The compiler seems slower probably because the structures are heavier to instanciate + we babylon.parse each and every expression that is considered javascript code. The code is then generated and reparsed via with. There are certainly ways to improve on this.

Feedbacks welcome !

[jeromew]

From @TimothyGu on October 30, 2016 1:17

For the with module, you can now either use the new (not-yet-published) https://github.com/pugjs/with, which takes a Babylon AST, or https://github.com/pugjs/babel-plugin-transform-with.

[jeromew]

From @ForbesLindesay on October 30, 2016 16:3

Thanks for doing all this. It may be a few weeks before I get time to look at it properly. I do think eventually it would be good to restrict unbuffered code to valid statements, but that transition is definitely going to be hard on users of pug, so we need to think about it carefully. We could start warning now, we could also support some kind of white-list of the most commonly used non-statements like - if (foo) { for a bit longer. My priority is getting to the stage where you never get an error from pug that doesn't tell you the line number/file etc. where the error is.

[jeromew]

@TimothyGu thanks for the info on the babel based with modules i'll look into that to use either of them.

@ForbesLindesay yes I think that we will be able to report line number/file of errors at compilation time + hopefully it should be possible to reach a jade->js sourcemap with close enough mapping.

Regarding unbuffered code, I think the implementation i found should give us the choice & an upgrade path. Currently it accepts locally invalid statements just as pug does but a boolean could be added to restrict compilation to locally valid statements only.

I believe the main culprits in the wild are if/else ifelseblocks andtry/catchblocks. But I stumbled into theitems.forEach` pattern in the pug tests cases.

I'll try to see how source maps can be generated and tested programmatically. If you have experience on this I am interested.

I'll probably fork pug to modify its package.json to point to my forked pug-code-gen so that it can be tested more easily.

[jeromew]

From @ForbesLindesay on October 30, 2016 17:7

I think babel-code-gen will need forking to add support for multiple files.

re forking pug. That makes sense. I think we should move most of pug into a mono-repo as it will make testing/releasing much safer and easier. I'd like to keep react-pug, then-pug, pug-cli, plugins etc. separate as it's very useful to have their issue & pull requests separate, but the pug-lexer, pug-parser etc. would probably be better in the same repo but separate npm packages.

[jeromew]

You can install a version of pug using babel AST using https://github.com/jeromew/pug
It uses the master branch of https://github.com/jeromew/pug-code-gen

With the help of @TimothyGu I used babel-plugin-transform-with to transform the AST instead of relying on with. All tests are passing.

I could not make is-expression-babylon to work because it seems the babylon stopped exporting the Parser - cf pugjs/is-expression-babylon#1

I am a bit worried and clueless at this stage regarding performance. I did not measure it precisely, but I do have more red and yellow tests with AST than with the previous method.

before:

• concatenate strings to build the source tree
• parse the source tree to find globals (with)
• build the final source string

after

• build the AST tree step by step (this means parsing all the js expressions)
• transform the AST tree to intercept globals (babel-plugin-transform-with)
• generate the source code

In my opinion, the perf difference should not be significant since with also has to parse the whole tree I suppose to find globals. I know @TimothyGu told me that babel-traverse is slower than babylon-walk but I am not enough versed into these to understand why at this stage.

I also thought that parsing small expressions would be faster than parsing the final javascript because there is less context to parse.

The problem I have is that at this stage, there is no code to "fix" the line/pos of all the AST nodes and that I suspect that this code is going to add more time penalties. We will need this nevertheless to generate source maps.

I add @alubbe here since you seem to know a lot about node optimisations and that maybe I am doing something wrong when I build the AST tree (a lot of array push for example).

[jeromew]

From @TimothyGu on October 31, 2016 15:59

babel-traverse is inherently slow because of all the features it supports -- paths, scopes, etc., while babylon-walk is just a bare-bones node-to-node walker. I don't think speed should be that much of a concern currently, but if needed we can profile it in the future.

[jeromew]

From @TimothyGu on October 31, 2016 16:6

And I was wondering what you meant by "fixing" line/pos of AST nodes. If you meant attaching line/pos of Pug source code so that sourcemaps could be generated, we need to add code column support to the parser first. I'm not terribly sure, but we might need to add support for tracking code spans in lexers (as currently only the beginning position is tracked).

[jeromew]

Yes I haven't looked into sourcemaps. I saw that we have a line/pos of the beginning position of pug tokens and was thinking about using that.

getExpression(expr)

would build an AST out of expr with line/pos in the expr string and I thought I would offset this with the line/pos given in the pug node (at least for js code)

but I haven't started looking into this. maybe this would not be sufficient for a useful sourcemap.

[jeromew]

From @ForbesLindesay on November 1, 2016 11:12

I think we will need end locations for tokens in the lexer. Once we have that it should be moderately simple, but I'm not sure.

[jeromew]

From @jbsulli on January 17, 2017 6:0

I would be interested in adding the end locations for for tokens in the lexer. Could be handy for some of the stuff I'm playing with and I would like to start getting more familiar with the pug internals. Would that be helpful? Or have you already started working on it?

[jeromew]

From @ForbesLindesay on January 17, 2017 8:47

That would be hugely helpful. Ideally it would be best to update the pug lexer and parser to match the Babylon, loc format.

[jeromew]

From @jbsulli on January 17, 2017 9:29

I like that idea. I've been working with Babylon some and would've suggested it. I'll keep you posted!

[jeromew]

From @jbsulli on January 17, 2017 16:13

I should clarify: What would be considered the end of tags/mixins/etc?

span#test.test(test=123) hello world
---^-------------------^-----------^
   1                   2           3

1. End of tag/mixin/etc name (my guess)
2. End of all properties/attributes
3. End of all contained content

My guess is just the end of the name as properties and content would be more in the domain of the parser. Just thought I'd check though.

[jeromew]

From @ForbesLindesay on January 17, 2017 16:13

This is an interesting point. I guess what we possibly ought to do is add Identifier and JavaScriptExpression and so on as AST types, then these can have a range and represent the keyword span, and each individual attribute.

In the lexer, the tag ends at position 1, because the stream of tokens does not encode the tree-structure.
I think once it gets to the parser, the Tag ast node is actually the whole thing (i.e. option 3).

[jeromew]

Copied from original issue: pugjs/pug-code-gen#21

[jeromew]

for info I migrated the babel AST version of pug-code-gen to monorepo on https://github.com/jeromew/pug

[jeromew]

For information, I started working on then-pug which was my original motivation for porting pug-code-gen to babel ast.

At this stage I use https://github.com/jeromew/pug for both the pug-code-gen babel port and a then-pug package in the monorepo. I will probably keep managing then-pug as a package in a fork of the monorepo if then-pug does not reach monorepo status. It makes things easier for tests.

currently I created a then-pug-standalone branch where the then-pug package uses an internal copy of pug-code-gen babel version. This was published on npm as then-pug 1.0.0 as it seems to be working as expected.

I am not yet satisfied with the pug-code-gen code organisation and method naming. There is room for improvement.

I am also waiting on a new babylon version now that parseExpression has been merged.

[jeromew]

@ForbesLindesay what is your current thinking on the rewrite of pug-code-gen using babel and direct AST generation ?

I have seen your work on the react generator which takes a different approach (I did not really dig into it yet) and was wondering how this all fits together in your view.

Is it worth maintaining my AST branch ? is there a target for switching the current pug-code-gen with an AST based solution ? Should I try and rewrite then-pug (which currently uses the AST branch) using an approach similar to pug-react ?). How does this fits with sourcemaps and the work that was done on token lineno/pos ?

Thanks for you help

[jeromew]

@ForbesLindesay, @jonathanong can you help me see what is the status of this issue ?

tldr:

• the idea was initiated by @ForbesLindesay (Nov 2015) that once pug would reach 2.0 it would be interesting to rewrite code-gen by directly producing babel AST instead of js strings.
• I wrote a first version of this (Feb 2017) that passes all the tests but did not have any feedback

how can I present you this work to see if this has any chance to reach upstream ? I would understand if you changed your mind and do not want to pursue this AST path.
I can spend some time on this in the following weeks if this is something you want to see go through, but it is not possible without your feedback because it changes code-gen too drastically.

[ForbesLindesay]

Hi @jeromew sorry I've been a bit lax about this. I think the time is probably right to attempt the switch. Can you create a "babel" branch on this repository and apply your changes to pug-code-gen on that branch, then create a pull request. I will then review it. If you can do this by 7th June, I can review it on the 8th June, I might have time before then, but I'm not sure.

[jeromew]

@ForbesLindesay I pushed the PR as #3019

[jeromew]

@ForbesLindesay Hello this is a ping on this old issue to know if you have any new direction / insights in mind for this.