[bug] Missed token for regular expressions

[EgorBu]

Hi,
I found that token information for node with regexp is missed.
Code:

function addResponsiveClass(str) {
  return str.replace(/\<img/g, '<img class="img-responsive"');
}

Let's visualize code without nodes with tokens:

function __________________(___) {
  return ___._______(/\<img/g, _____________________________);
}

If we check the node with /\<img/g we will see:

internal_type: "RegExpLiteral"
properties {
  key: "flags"
  value: "g"
}
properties {
  key: "internalRole"
  value: "arguments"
}
properties {
  key: "pattern"
  value: "\\<img"
}
start_position {
  offset: 283
  line: 14
  col: 22
}
end_position {
  offset: 291
  line: 14
  col: 30
}
roles: EXPRESSION
roles: LITERAL
roles: REGEXP
roles: CALL
roles: ARGUMENT

I think this node should contain token with /\<img.

[dennwc]

Blocked by bblfsh/sdk#318

[dennwc]

Can confirm that it's related to an issue above. But if we can solve it right now if we duplicate some information in the node.

For the context, the native AST node contains the following fields:

• raw = /\<img/g (both regexp and the flag)
• pattern = \<img (regexp only, unescaped)
• flags = g (flag only)

Right now we set @token to the regexp pattern (\<img) and leave flag (g) as a separate field. And it fails ML expectations.

Instead, I will set the @token to the full regexp, while preserving pattern and flag fields separately.