Modernize bundle: rewrite grammar and add new commands/snippets.

[savetheclocktower]

Apologies up front; this is a large PR with lots of disparate changes. If some of this stuff isn’t wanted, I’m happy to submit another PR with only the consensus changes.

Also: I applied these changes to the existing grammar as a delta and then used the “Export Bundle” function in the bundle editor, which resulted in some of the existing commands and snippets having new filenames. So they show up as deletions and additions instead of changes. If there’s a better process for making changes to a managed bundle, let me know.

Here’s a summary of what’s in this PR:

Rewritten grammar

Inspired by the suggestion in #51, I started with the “JavaScript Babel” grammar originally written for Atom (or, rather, one of the several ports to TextMate, though I forget which).

This gets us syntax highlighting support for all new ES2015 syntax, plus JSX, plus some syntax that’s not yet official but on the standards track (like decorators).

I fixed a few things:

• places where the grammar tried to add two space-separated scopes in a single capture (which causes some really subtle and frustrating problems);
• places where the scope names went against TM conventions (anything showing up red in the bundle editor);
• removed rules for Flow, which I don’t think is anywhere near common enough to warrant inclusion in a generic JavaScript grammar.

I left in the rules for JSX because of how commonplace it is, and so that I wouldn’t have to write an additional grammar that wrapped this one in order to add JSX (though that might be an eventual goal).

I also added back some of the things in the existing grammar, like the extensive list of web APIs that are recognized and highlighted as support.

Rewritten “Documentation for Word” command

I liked the approach of the existing command, but never used it because of how little it actually recognized — only operators, keywords, and tokens like Array and RegExp.

I wrote a script to crawl selected sections of the MDN documentation and generate a YAML file organized by token. For instance, indexOf has entries for both String.prototype.indexOf and Array.prototype.indexOf. Hitting ^H inside a token will either take you straight to the single relevant match (90% of the time) or, in the case of multiple matches, call TextMate::UI::menu to let the user pick which one they want.

If the token isn’t found in the index, the command will open a page of MDN search results for the token.

The PR contains this YAML file but not the script used to generate it, since an end user wouldn't have to run it (and since it depends on Ruby 2 and the Nokogiri gem). The script should probably go somewhere, though, so that’s a topic for discussion.

New “Run in Node” command

It was quite easy (and long past due) to write an analog to the Ruby “Run Script” command. The command does include some special-case logic for NVM so that .nvmrc files are respected if one is present in the same directory as the file you’re executing. This only comes into play if a user has NVM installed, and only if TM_NODE is not explicitly set.

New snippets

I added a few snippets I thought deserved inclusion:

• a snippet for console.debug to go along with the other logging snippets — all of which are now in their own sub-menu named "Logging";
• snippets for require (CommonJS) and import (ES Module syntax);
• snippets for new Promise and a chained .then on a Promise instance.

New settings

I added a couple of settings:

• one for disabling automatic indent/unindent within template strings;
• two for ensuring functions and methods are included in the symbol list. (Class methods get prepended with #.)

[savetheclocktower]

Also: if the new grammar is landed, it’ll obviate the change I made in PR #62.

[patrickrgaffney]

I have been using this for a few days at work — it's a great improvement over the current grammar. It would also fix a couple of open issues (#60 at least).

It would probably make more sense to have the JSX rules in a separate grammar (they only really make sense when working with React, it's not a formal part of the JS language). It could probably be implemented as a separate grammar via an injection selector (described here)

[savetheclocktower]

Yeah, I've used injection selectors in the past. Based on my recollection, their main weakness is that they can only inject themselves into scopes that have been defined with begin and end patterns, rather than match patterns. Which is tough for a construct that can be placed nearly anywhere.

That said, I didn't actually take a stab at it, so if I get time over the holidays I'll try to put JSX in its own grammar. It may not be as tricky as I remember.

[savetheclocktower]

I take it back! That was way easier than I expected. Prepending L: to the injection selector is the trick I was not aware of. I'll definitely update this PR soon with the results.

[savetheclocktower]

Question, @patrickrgaffney: are you envisioning that the JSX grammar be something that people opt into, or something that just works? It seems like the best approach for an injected grammar is to hide it from the grammar menu, and have it just automatically put itself into all source.js files, but in that case I'm not sure what we gain by isolating JSX within its own grammar in the first place.

[savetheclocktower]

Bump.

[everydayscience]

Great work on updating the bundle.

I’ve been playing with it locally and have discovered that the arrow function syntax seems to break if you have new lines inside the arguments parentheses.

For example, the following will not be recognised as an arrow function:

const someFunction = (
anArgument,
anotherArgument
) => 'a return value';

[desandro]

Thank you for this updated bundle! This resolved several issues I was seeing with highlighting basic stuff like constructors and support methods. I'd like to see this merged to master 👍

[pdokas]

I just installed this as well and it’s working out very nicely!

[hovsater]

This is truly wonderful work, @savetheclocktower. I think we should make an effort to get these changes merged!

[ogallagher]

It looks like // TODO something comments are no longer highlighted like /* TODO something */?

[ogallagher]

I use jsdoc-toolkit for highlighting jsdoc comments, but it seems to be incompatible with these changes?

master

pull/63

[ogallagher]

To fix my previous comment, I opened a pull req to recover the jsdoc keywords pattern in comment blocks.