html/template: add support for template strings (backticks)

[gopherbot]

by opennota:

ES6 specifies a new language feature called "Template Strings" (often also
referred to as "Quasi Literals" alongside multi-line strings and others). This
allows to execute arbitrary JavaScript code without using parenthesis but
back-ticks instead. Inside back-tick delimited strings, placeholders such as
${} can wrap executable code.

http://play.golang.org/p/nBEneuxHNj

If you open the output of the above program in a modern browser (e.g., recently released
Mozilla Firefox 34 supports template strings), it will happily execute alert(1).

References:

https://people.mozilla.org/~jorendorff/es6-draft.html
https://html5sec.org/#140
https://html5sec.org/#141

[ianlancetaylor]

Comment 1:

Labels changed: added repo-main, release-go1.5.

[robpike]

Comment 2:

Just what JavaScript needed, a new way to execute arbitrary code on the client machine.

Owner changed to mikesamuel.

Status changed to Accepted.

[gopherbot]

Comment 3 by mikesamuel:

I really should have seen that coming since I wrote the spec strawman:
https://code.google.com/p/js-quasis-libraries-and-repl/
This doesn't actually introduce a new execution vector for client-side code since the
backticks are significant inside a block of JavaScript code, not inside HTML.  Automatic
interpretation of '{{' inside <template> elements does that.  (sigh)
Will fix.

[mcfedr]

Might also be an issue inside of JS attributes, as in this demo: https://play.golang.org/p/iahSK9oqLc - pops up alert in modern browsers

[mvdan]

#29406 was about malformed template output since backticks aren't parsed properly. I've re-titled this issue to be about the overall support of this ES6 feature, and closed the newer issue as a duplicate.

I'm marking it as NeedsDecision, just in case anyone disagrees with adding this feature. It's part of the JS standard, so as long as html/template follows that, it seems to me like this issue should be fixed.

[hundt]

I'm not sure if anyone else is working on this, but I have given it some thought and I thought I'd share in case anyone else has some insight.

First, some notes about template literals:

• They are enclosed by backticks.
• They can contain "placeholders", which are marked by "${" on the left side and "}" on the right side, and which can contain any valid JS expression.
• "Valid JS expressions" include template literals and things containing them, so they can be nested: `${`${`${1}`}`}` is valid. And this is a common use case, not an obscure edge case.
• The non-placebolder parts of the template literals are treated as string literals.

I am not super-familiar with this codebase but to me there are two main issues that need to be solved:

1. Detecting when we are in a template literal or placeholder expression in a Go template, so that we can escape properly, and also so that we don't get confused and parse the rest of the Go template incorrectly (as in #29406)
2. Detecting when the data passed to Execute will be interpreted as a placeholder, allowing arbitrary code execution (as in the example given when this issue was opened).

I think if we can solve problem 1 then problem 2 is straightforward: simply replace "$" with "\x24" when inserting into a template-literal context. In the regular javascript context (including the placeholder context) any data with a backtick would already have quotes wrapped around it so it would not be considered a template literal.

Solving problem 1 is a harder given the approach of the existing code, which steps through the characters of the JS code but doesn't really parse the JS. In the code, the state of being in regular Javascript code is called stateJS. Let's suppose we add a new state stateTL for being in a template literal. Then "`" marks a transition stateJS -> stateTL or stateTL -> stateJS. And "${" also marks a transition stateTL -> stateJS. But "}" may mark a transition from stateJS -> stateTL (if it is marking the end of a placeholder) or it might just be the end of a code block. So we will need to know whether our current stateJS state is part of a placeholder inside of a template literal or not. And since template literals can nest we'll need to know how many levels deep we are.

I implemented the above as my initial solution, but then I realized it still isn't quite right, because the placeholder expression could contain "}"s. For example, `${function(){return 1}()}` is a valid template literal. So when inside a placeholder expression we also need to keep track of how many blocks deep we are, so that we know when the expression finally ends. And, since we could be multiple template literals deep, I suppose we need a stack of block depths representing how deep we are inside each placeholder expression along the way. Otherwise, given the expression `${function(){if (1>0) {return `${1}`} else {return 2}}()}`, when we exit out of the innermost template literal ("`${1}`") into the enclosing placeholder expression, we won't know if the following "}" closes that placeholder expression or not.

At this point it seems to me that I am thinking of a level of complexity beyond how the JS-handing code currently works. So I'm wondering if I'm missing a simpler solution, or if these template literals complicate the situation enough that it is simply necessary to have a more complex Javascript "parser" for html/template to work with them.

[FiloSottile]

/cc @empijei

[empijei]

Thanks @hundt for your analysis.

It seems to me that fixing this issue would require this package to implement a JS parser to keep track of the current context. If not a fully-fledged JS parser it would at least need to correctly understand the context-switch between Template Literals and JavaScript context.

While this is possible, it would complicate the current logic more than I'm comfortable with, and seems very tricky to get right. I played with this a little and stuff like the following line came up:

var tmpl = `"${a="`${"}}`;

My opinion would be to not support this and disallow JS template literals. This would need the documentation to be updated accordingly.

I have some ideas on how to do this:

1. Accept backtick-delimited strings just to have multiline strings or easier quotes like Go does, and encode ${ to prevent execution;
2. Bail out and warn users that backticks are not supported at parse time;
3. Do not sanitize, nor change change anything that is in a template literal, but this is trickier and more dangerous and still needs decision on what to do when a variable is interpolated in such context;

I would honestly already update the docs to say that we currently do not support template literals.

We also need to decide what to do for cases like

alert`1`

Which are valid JS function calls.

@mikesamuel do you have a different opinion?

@hundt and @eternal-flame-AD what is the use case for JS template literals inside html templates? (I can guess hypothetical ones but I'm interested in your concrete one if you can publish it)

[hundt]

A few notes:

I played with this a little and stuff like the following line came up:

var tmpl = `"${a="`${"}}`;

I think this kind of thing is pretty well handled by the internal state machine once it understands the backticks, because it already knows that characters inside quotes lose the special meaning they have outside quotes. I tried this input with my initial implementation that I mentioned above, with some logging to indicate state transitions, and it did the right thing:

Parse "var tmpl = `": {stateJSTLStr delimNone urlPartNone jsCtxRegexp jsTLDepth=1 attrNone elementScript <nil>}
Parse ""${": {stateJS delimNone urlPartNone jsCtxDivOp jsTLDepth=1 attrNone elementScript <nil>}
Parse "a="": {stateJSDqStr delimNone urlPartNone jsCtxRegexp jsTLDepth=1 attrNone elementScript <nil>}
Parse "`${"": {stateJS delimNone urlPartNone jsCtxDivOp jsTLDepth=1 attrNone elementScript <nil>}
Parse "}": {stateJSTLStr delimNone urlPartNone jsCtxRegexp jsTLDepth=1 attrNone elementScript <nil>}
Parse "}`": {stateJS delimNone urlPartNone jsCtxDivOp jsTLDepth=0 attrNone elementScript <nil>}

Here stateJSTLStr is the new state I added for being in a template literal string, and jsTLDepth indicates how many template literals deep you are.

We also need to decide what to do for cases like

alert`1`

Which are valid JS function calls.

I'm not sure it would be necessary to explicitly handle this. As long as backticks are implemented, I believe the current code will correctly understand that the "alert" part is in the normal Javascript context and we leave that context with the backtick.

@hundt and @eternal-flame-AD what is the use case for JS template literals inside html templates? (I can guess hypothetical ones but I'm interested in your concrete one if you can publish it)

Honestly in my case it was just a case of being lazy and using an embedded script with rendering logic inside of it on a page that also used html/template to inject in some server-side variables. My experience with this issue and my later reading of #27926 has convinced me that this was a bad approach, and I refactored my code to put substantially all of the Javascript into separately loaded modules which no longer pass through any server-side templating, which is what I probably should have done in the first place.

Given that it is probably not good practice to do what I had been doing, in my opinion it would be reasonable not to support HTML templates with JS template literals (and fail when it encounters them), or to support it only with template literals formatted in the subset of possible ways that html/template can understand (provided it can reliably detect and error out when it encounters an unsupported situation). To me the biggest problem is that the API currently promises "safe" HTML but does not deliver it. Better to simply fail in that situation.

In case it is of interest: I gathered that Google's closure-templates library is well-regarded, so I checked what they do. Just like html/template, they do a character-by-character "context" analysis of Javascript that falls short of fully parsing it. Their approach to template literals appears to be identical to the initial implementation I described above, including the failure to deal with braces inside ${. However, their template parser fails at the parsing stage of such a template, either due to an additional bug or by design (I'm not sure which).

[empijei]

my initial implementation that I mentioned above

Do you want to share a link to the patched code?

To me the biggest problem is that the API currently promises "safe" HTML but does not deliver it. Better to simply fail in that situation.

I agree. Plus I see little to no reason to nest Go and JS templates.

The solution I'm more inclined to implement and document is to just have a stateJSTLStr and encode $`\ and other potentially dangerous characters (this would probably require some thoughts on what to encode/filter and how).

@hundt if we can get to an agreement on this would you like to work on a CL to address it?

[hundt]

my initial implementation that I mentioned above

Do you want to share a link to the patched code?

Ah, sorry, wasn't sure the right way to share it without requesting a PR. Here's a patch, does that work for you? https://gist.github.com/hundt/fc27e649a01722a8466987b0f102fe5d

The solution I'm more inclined to implement and document is to just have a stateJSTLStr and encode $`\ and other potentially dangerous characters (this would probably require some thoughts on what to encode/filter and how).

I think if we are going to support template literals then we would need to support ${} and nested template literals (in the HTML template, not in the input variables), since my impression is that many (most?) practical uses of template literals involve nesting (because that is how you do conditionals and loops). I believe we can do that without making my implementation much more complicated if we just fail when we encounter { (without the $) inside ${}, to avoid the problem of not knowing when the ${} ends. (I'm assuming there's a way to error out and abort when we see certain characters instead of doing a state transition but I haven't spent enough time looking at the code to be sure.)

@hundt if we can get to an agreement on this would you like to work on a CL to address it?

I'm interested but I will likely not be available in the short term. After another day or so I will likely have little time to work on this for at least two weeks. But if you don't mind waiting I can probably pick it up then.

[hundt]

In case it helps illustrate some of the issues, here are some test cases indicating potential failure modes that allow code injection:

https://play.golang.org/p/61h5vn53A4H

The current code fails the first, second, and fourth tests because it does not understand template literals.

My patch fails the fifth test because it understands template literals but not braces inside ${}

[empijei]

Thanks for this.

To be honest I'd be more in favor of not supporting template literals at all instead of just partially supporting them (as much as you patch cover a good share of cases, aborting on [^$]{ seems like a hack and counts as partial support).

I was proposing to drop all ${} expressions to have some backtick-delimited-multiline-strings with easier quoting but not templating features (after all we are in a template package).

If this is not a common use case and wouldn't be useful I would remove TL completely. The approach would then be to just filter them or bail out when we see a backtick outside of strings.

[hundt]

Ah okay, I misunderstood your proposal. I'm not familiar enough with how browsers parse things to know how feasible it is to "drop all ${} expressions" in a reliable enough way that nothing could sneak in, but assuming it can be done that seems okay to me. I would not use such a feature though; I think there is not much point in using template literals if they can't be templates.

I think it may be better not to support template literals at all (i.e. fail when we encounter them). #27926 has kind of convinced me that anything that discourages people from passing JS of any complexity through html/template might be a good idea. To quote from that proposal:

JS and CSS parsing is error-prone. JS and CSS are rapidly evolving languages that our parser might not always handle correctly; layering parsers for these two languages on top of our mixed HTML-template language parser introduces more complexity and points of failure to the package.

This is an example of an issue that did not exist when html/template was written, arose years later, and then continued on unfixed for more than four additional years, potentially leaving users vulnerable during that time. So I think it is good evidence in favor of the argument quoted, and against encouraging people to rely on html/template to understand their Javascript. So for that reason I am tentatively in favor of just not adding support for new Javascript features like this one. (Edit: but I do think we should add code to detect them, or anything that looks like them, and fail the template compilation.)

[empijei]

You have a good point.

+1 on bailing out if we see a template literal.

If @mikesamuel agrees we can start working on it.

[mikesamuel]

I'm not familiar enough with how browsers parse things to know how feasible it is to "drop all ${} expressions"

It can be done purely based on lexical analysis.
There are a few oddities because of JS is irregular around '/'.

let myRegExp = / `${2}`.length /   // No interpolation here.
2;
let myNum = 10 / `${2}`.length /   // An interpolation here
2;

i ++
   / `${2}`.length /  // Interpolation here
['index']

i =
++ / `${2}`.length /  // No interpolation here
['index']

${2} is not useful in regular expression literals; it means zero-width match of end-of-input twice.

So matching ${ outside comments and single- or double-quoted strings that are not preceded by an odd-number of \s will reliably catch all interpolations.

#27926 has kind of convinced me that anything that discourages people from passing JS of any complexity through html/template might be a good idea.

I agree. I included it because (JS was very stable at that time) and I wanted to enable

<script>intiialize({{ .InitialClientSideState }})</script>

If @mikesamuel agrees we can start working on it.

Fwiw, +1

[hundt]

#27926 has kind of convinced me that anything that discourages people from passing JS of any complexity through html/template might be a good idea.

I agree. I included it because (JS was very stable at that time) and I wanted to enable

<script>intiialize({{ .InitialClientSideState }})</script>

Yeah, that seems very useful! My issue as a user who was not familiar with the codebase was that it was not clear to me that I should stick to very simple things like that and not try to pass any non-trivial scripts through html/template.

If @mikesamuel agrees we can start working on it.

Fwiw, +1

Just to be clear, you're talking about failing for any JS with template literals, right? And the earlier part of your comment (about interpolation) was for my edification (thanks!) but not relevant to the solution that would be implemented?