Template string tokens are strange

[nzakas]

The way that template string tokens are generated is pretty strange. This code:

var foo = `hi${bar}`;

Is tokenized as:

[
    {
        "type": "Keyword",
        "value": "var"
    },
    {
        "type": "Identifier",
        "value": "foo"
    },
    {
        "type": "Punctuator",
        "value": "="
    },
    {
        "value": "`hi${"
    },
    {
        "type": "Identifier",
        "value": "bar"
    },
    {
        "type": "Punctuator",
        "value": "}"
    },
    {
        "type": "Punctuator",
        "value": ";"
    }
]

So the first part of the template string is represented by a token without a type, and then the rest are just regular tokens, completely losing the end of the template string.

[nzakas]

Working on this.

[nzakas]

Not quite fixed. The code var a; console.log(${a}, \"a\"); still outputs this:

[ { type: 'Identifier',
    value: 'console',
    range: [ 7, 14 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Punctuator',
    value: '.',
    range: [ 14, 15 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Identifier',
    value: 'log',
    range: [ 15, 18 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Punctuator',
    value: '(',
    range: [ 18, 19 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Template',
    value: '`${',
    range: [ 19, 22 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Identifier',
    value: 'a',
    range: [ 22, 23 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Punctuator',
    value: '}',
    range: [ 23, 24 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Punctuator',
    value: ',',
    range: [ 25, 26 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'String',
    value: '"a"',
    range: [ 27, 30 ],
    loc: { start: [Object], end: [Object] } },
  { type: 'Punctuator',
    value: ')',
    range: [ 30, 31 ],
    loc: { start: [Object], end: [Object] } } ]

[nzakas]

Followup: This was actually a problem with using parse(), not when using tokenize(), since they use different logic.