Support HTML and CSS syntax in template strings suggesting those formats

[jrburke]

Tagged template strings allow embedding HTML and CSS in a JavaScript file, but allow a safe way to escape the values provided by the JavaScript context, and allows a tool to statically verify the use of this pattern.

Since tagged template strings provide a name for the function used to process the template string, that name can be used to target using HTML or CSS syntax highlighting within a template string.

This pull request enables tagged string use whose tag function name ends in html to specify HTML syntax highlighting inside the string. Same with css.

There are cases where the interpolated_js loses out to the HTML or CSS highlighting, like if ${x} is used in an HTML attribute, but I believe that is just a limit of regexp-based parsers. If HTML/CSS is in the string, it still seems better to favor the larger format in the string to allow the most gain for visually parsing the contents.

I am new to this syntax highlighting format, so I am not sure if I followed best practice for it. For instance, perhaps it is better to extract the constant.character.escape.js as a separate thing and just refer to it in each of the template string-related blocks. The constant.character.escape.js used for " and ' strings are slightly different between them though, so not sure if that was intentional or if both ' " and the template string sections could use the same constant.character.escape.js, as specified for the original template string definition.

This works for me locally, and a couple of very simple tests were added.

[50Wliu]

Hey @jrburke, sorry for the delayed review of this.

I have a couple of issues with this - you mentioned that the interpolation will sometimes lose out to the HTML/CSS styling: will that break the rest of the formatting, or will it gracefully recover?
Also, it's not necessary for HTML/CSS templates to begin with html or css - maybe it could be called htmlify or style - who knows? I think that it might be annoying for only those specific patterns to receive highlighting, but again this seems like a limitation of regex. What do you think should be done about those cases?

[jrburke]

@50Wliu:

I have a couple of issues with this - you mentioned that the interpolation will sometimes lose out to the HTML/CSS styling: will that break the rest of the formatting, or will it gracefully recover?

It does seem to recover, and interpolated_js does seem to apply for cases where the HTML/CSS highlighting does not take priority. Here are some screenshots I used in testing, using a solarized light theme:

HTML:

CSS:

Also, it's not necessary for HTML/CSS templates to begin with html or css - maybe it could be called htmlify or style - who knows? I think that it might be annoying for only those specific patterns to receive highlighting, but again this seems like a limitation of regex. What do you think should be done about those cases?

I believe syntax highlighted tagged template strings this will become similar to mime types, where there are conventions used to get the desired effect. Out of the possible parts of a JS function name to use for indicating the string type, it seems using the suffix, ending part of the function name makes the most sense: it is closest to the string value, and is a clearer regexp to use for syntax highlighting, no need to account for the full character set allowed in function names. If a developer wanted to get this syntax highlighting, but the tag function name was not the correct form, it is easy enough to create a local variable like var html = htmlify then use the html for the tagged template use.

[50Wliu]

The escapeHTML highlighting looks incorrect to me - either the whole thing should be highlighted or nothing. Right now only HTML is.

[jrburke]

I fixed the escapeHTML highlighting. I understand the 'beginCaptures' better now, and I added some parens for the regexps, and fixed the \w to be \\w since it is in a string. The function name is now recognized as a function name.

I rebased over latest master and squashed the commits so that this pull request is just one commit, but here is the diff from the previous version of this pull request:

diff --git a/grammars/javascript.cson b/grammars/javascript.cson
index 0487808..5b8eb77 100644
--- a/grammars/javascript.cson
+++ b/grammars/javascript.cson
@@ -1009,9 +1009,11 @@
         ]
       }
       {
-        'begin': '(\w+)?(html|HTML|Html)`'
+        'begin': '((\\w+)?(html|HTML|Html))(`)'
         'beginCaptures':
           '0':
+            'name': 'entity.name.function.js'
+          '4':
             'name': 'punctuation.definition.string.begin.js'
         'end': '`'
         'endCaptures':
@@ -1032,9 +1034,11 @@
         ]
       }
       {
-        'begin': '(\w+)?(css|CSS|Css)`'
+        'begin': '((\\w+)?(css|CSS|Css))(`)'
         'beginCaptures':
           '0':
+            'name': 'entity.name.function.js'
+          '4':
             'name': 'punctuation.definition.string.begin.js'
         'end': '`'
         'endCaptures':

And for the tests:

diff --git a/spec/javascript-spec.coffee b/spec/javascript-spec.coffee
index 3110049..4a5fc98 100644
--- a/spec/javascript-spec.coffee
+++ b/spec/javascript-spec.coffee
@@ -488,24 +488,26 @@ describe "Javascript grammar", ->
   describe "ES6 tagged HTML string templates", ->
     it "tokenizes them as strings", ->
       {tokens} = grammar.tokenizeLine('html`hey <b>${name}</b>`')
-      expect(tokens[0]).toEqual value: 'html`', scopes: ['source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.begin.js']
-      expect(tokens[1]).toEqual value: 'hey <b>', scopes: ['source.js', 'string.quoted.template.html.js']
-      expect(tokens[2]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
-      expect(tokens[3]).toEqual value: 'name', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source']
-      expect(tokens[4]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
-      expect(tokens[5]).toEqual value: '</b>', scopes: ['source.js', 'string.quoted.template.html.js']
-      expect(tokens[6]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.end.js']
+      expect(tokens[0]).toEqual value: 'html', scopes: [ 'source.js', 'string.quoted.template.html.js', 'entity.name.function.js' ]
+      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.html.js', 'entity.name.function.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[2]).toEqual value: 'hey <b>', scopes: ['source.js', 'string.quoted.template.html.js']
+      expect(tokens[3]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[4]).toEqual value: 'name', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source']
+      expect(tokens[5]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[6]).toEqual value: '</b>', scopes: ['source.js', 'string.quoted.template.html.js']
+      expect(tokens[7]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.end.js']

   describe "ES6 tagged CSS string templates", ->
     it "tokenizes them as strings", ->
       {tokens} = grammar.tokenizeLine('css`.highlight { border: ${borderSize}; }`')
-      expect(tokens[0]).toEqual value: 'css`', scopes: ['source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.begin.js']
-      expect(tokens[1]).toEqual value: '.highlight { border: ', scopes: ['source.js', 'string.quoted.template.css.js']
-      expect(tokens[2]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
-      expect(tokens[3]).toEqual value: 'borderSize', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source']
-      expect(tokens[4]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
-      expect(tokens[5]).toEqual value: '; }', scopes: ['source.js', 'string.quoted.template.css.js']
-      expect(tokens[6]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.end.js']
+      expect(tokens[0]).toEqual value: 'css', scopes: [ 'source.js', 'string.quoted.template.css.js', 'entity.name.function.js' ]
+      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.css.js', 'entity.name.function.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[2]).toEqual value: '.highlight { border: ', scopes: ['source.js', 'string.quoted.template.css.js']
+      expect(tokens[3]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[4]).toEqual value: 'borderSize', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source']
+      expect(tokens[5]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[6]).toEqual value: '; }', scopes: ['source.js', 'string.quoted.template.css.js']
+      expect(tokens[7]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.end.js']

   describe "ES6 class", ->
     it "tokenizes class", ->

Here is how it looks now with the light solarized theme:

[50Wliu]

Ok, just a few more questions. Can there be spaces in between the function and the backtick? Eg

html   `hi`

And can you also please add a test for something like escapeHTML?

[jrburke]

1. Space between the function name and template string: I checked the spec, and it does not specify the name must be directly next to the backtick, and I tested in Chrome and Firefox, and indeed it is treated like a function call, where function run() {} can be called as run() or run () or run\n().

So I updated the regexp to allow \s* before the backtick. I also adjusted the beginCaptures to use '1' instead of '0' for the entity.name.function.js. This excludes the spaces, and it matches what I see when I inspect run () in atom with the developer tools.

Both run\n() and `html\n`` lead to the match not matching, so this change behaves as well as the plain function calls in that respect.

2. I added more tests to include ecapeHTML, one with spaces and equivalent tests for the CSS case.

The display in the editor is the same as before, just now works with spaces between the function name and the backtick, so I am not including new screenshots.

I rebased over latest master and flattened commits, but here is the diff between last state:

diff --git a/grammars/javascript.cson b/grammars/javascript.cson
index 5b8eb77..4a34520 100644
--- a/grammars/javascript.cson
+++ b/grammars/javascript.cson
@@ -1009,9 +1009,9 @@
         ]
       }
       {
-        'begin': '((\\w+)?(html|HTML|Html))(`)'
+        'begin': '((\\w+)?(html|HTML|Html))\\s*(`)'
         'beginCaptures':
-          '0':
+          '1':
             'name': 'entity.name.function.js'
           '4':
             'name': 'punctuation.definition.string.begin.js'
@@ -1034,9 +1034,9 @@
         ]
       }
       {
-        'begin': '((\\w+)?(css|CSS|Css))(`)'
+        'begin': '((\\w+)?(css|CSS|Css))\\s*(`)'
         'beginCaptures':
-          '0':
+          '1':
             'name': 'entity.name.function.js'
           '4':
             'name': 'punctuation.definition.string.begin.js'
diff --git a/spec/javascript-spec.coffee b/spec/javascript-spec.coffee
index 4a5fc98..6938bb2 100644
--- a/spec/javascript-spec.coffee
+++ b/spec/javascript-spec.coffee
@@ -489,7 +489,7 @@ describe "Javascript grammar", ->
     it "tokenizes them as strings", ->
       {tokens} = grammar.tokenizeLine('html`hey <b>${name}</b>`')
       expect(tokens[0]).toEqual value: 'html', scopes: [ 'source.js', 'string.quoted.template.html.js', 'entity.name.function.js' ]
-      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.html.js', 'entity.name.function.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.begin.js' ]
       expect(tokens[2]).toEqual value: 'hey <b>', scopes: ['source.js', 'string.quoted.template.html.js']
       expect(tokens[3]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
       expect(tokens[4]).toEqual value: 'name', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source']
@@ -497,11 +497,36 @@ describe "Javascript grammar", ->
       expect(tokens[6]).toEqual value: '</b>', scopes: ['source.js', 'string.quoted.template.html.js']
       expect(tokens[7]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.end.js']

+  describe "ES6 tagged HTML string templates with expanded function name", ->
+    it "tokenizes them as strings", ->
+      {tokens} = grammar.tokenizeLine('escapeHTML`hey <b>${name}</b>`')
+      expect(tokens[0]).toEqual value: 'escapeHTML', scopes: [ 'source.js', 'string.quoted.template.html.js', 'entity.name.function.js' ]
+      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[2]).toEqual value: 'hey <b>', scopes: ['source.js', 'string.quoted.template.html.js']
+      expect(tokens[3]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[4]).toEqual value: 'name', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source']
+      expect(tokens[5]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[6]).toEqual value: '</b>', scopes: ['source.js', 'string.quoted.template.html.js']
+      expect(tokens[7]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.end.js']
+
+  describe "ES6 tagged HTML string templates with expanded function name and white space", ->
+    it "tokenizes them as strings", ->
+      {tokens} = grammar.tokenizeLine('escapeHTML   `hey <b>${name}</b>`')
+      expect(tokens[0]).toEqual value: 'escapeHTML', scopes: [ 'source.js', 'string.quoted.template.html.js', 'entity.name.function.js' ]
+      expect(tokens[1]).toEqual value: '   ', scopes: [ 'source.js', 'string.quoted.template.html.js' ]
+      expect(tokens[2]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[3]).toEqual value: 'hey <b>', scopes: ['source.js', 'string.quoted.template.html.js']
+      expect(tokens[4]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[5]).toEqual value: 'name', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source']
+      expect(tokens[6]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.html.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[7]).toEqual value: '</b>', scopes: ['source.js', 'string.quoted.template.html.js']
+      expect(tokens[8]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.html.js', 'punctuation.definition.string.end.js']
+
   describe "ES6 tagged CSS string templates", ->
     it "tokenizes them as strings", ->
       {tokens} = grammar.tokenizeLine('css`.highlight { border: ${borderSize}; }`')
       expect(tokens[0]).toEqual value: 'css', scopes: [ 'source.js', 'string.quoted.template.css.js', 'entity.name.function.js' ]
-      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.css.js', 'entity.name.function.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.begin.js' ]
       expect(tokens[2]).toEqual value: '.highlight { border: ', scopes: ['source.js', 'string.quoted.template.css.js']
       expect(tokens[3]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
       expect(tokens[4]).toEqual value: 'borderSize', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source']
@@ -509,6 +534,31 @@ describe "Javascript grammar", ->
       expect(tokens[6]).toEqual value: '; }', scopes: ['source.js', 'string.quoted.template.css.js']
       expect(tokens[7]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.end.js']

+  describe "ES6 tagged CSS string templates with expanded function name", ->
+    it "tokenizes them as strings", ->
+      {tokens} = grammar.tokenizeLine('escapeCSS`.highlight { border: ${borderSize}; }`')
+      expect(tokens[0]).toEqual value: 'escapeCSS', scopes: [ 'source.js', 'string.quoted.template.css.js', 'entity.name.function.js' ]
+      expect(tokens[1]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[2]).toEqual value: '.highlight { border: ', scopes: ['source.js', 'string.quoted.template.css.js']
+      expect(tokens[3]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[4]).toEqual value: 'borderSize', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source']
+      expect(tokens[5]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[6]).toEqual value: '; }', scopes: ['source.js', 'string.quoted.template.css.js']
+      expect(tokens[7]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.end.js']
+
+  describe "ES6 tagged CSS string templates with expanded function name and white space", ->
+    it "tokenizes them as strings", ->
+      {tokens} = grammar.tokenizeLine('escapeCSS   `.highlight { border: ${borderSize}; }`')
+      expect(tokens[0]).toEqual value: 'escapeCSS', scopes: [ 'source.js', 'string.quoted.template.css.js', 'entity.name.function.js' ]
+      expect(tokens[1]).toEqual value: '   ', scopes: [ 'source.js', 'string.quoted.template.css.js' ]
+      expect(tokens[2]).toEqual value: '`', scopes: [ 'source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.begin.js' ]
+      expect(tokens[3]).toEqual value: '.highlight { border: ', scopes: ['source.js', 'string.quoted.template.css.js']
+      expect(tokens[4]).toEqual value: '${', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[5]).toEqual value: 'borderSize', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source']
+      expect(tokens[6]).toEqual value: '}', scopes: ['source.js', 'string.quoted.template.css.js', 'source.js.embedded.source', 'punctuation.section.embedded.js']
+      expect(tokens[7]).toEqual value: '; }', scopes: ['source.js', 'string.quoted.template.css.js']
+      expect(tokens[8]).toEqual value: '`', scopes: ['source.js', 'string.quoted.template.css.js', 'punctuation.definition.string.end.js']
+
   describe "ES6 class", ->
     it "tokenizes class", ->
       {tokens} = grammar.tokenizeLine('class MyClass')

[50Wliu]

Awesome, thanks for this!