var jsTokens = require("js-tokens") var jsString = "var foo=opts.foo;\n..." jsString.match(jsTokens) // ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
npm install js-tokens
var jsTokens = require("js-tokens")
A regex with the
The next match is always directly after the previous.
var token = jsTokens.matchToToken(match)
match returned by
jsTokens.exec(string), and returns a
String, value: String} object. The following types are available:
Multi-line comments and strings also have a
closed property indicating if the
token was closed or not (see below).
Comments and strings both come in several flavors. To distinguish them, check if
the token starts with
Names are ECMAScript IdentifierNames, that is, including both identifiers and keywords. You may use is-keyword-js to tell them apart.
Whitespace includes both line terminators and other whitespace.
For example usage, please see this gist.
Invalid code handling
Unterminated multi-line comments are also still matched as comments. They simply go on to the end of the input.
Unterminated regex literals are likely matched as division and whatever is inside the regex.
Invalid ASCII characters have their own capturing group.
Invalid non-ASCII characters are treated as names, to simplify the matching of names (except unicode spaces which are treated as whitespace).
Regex literals may contain invalid regex syntax. They are still matched as regex literals. They may also contain repeated regex flags, to keep the regex simple.
Strings may contain invalid escape sequences.
You may compare jsTokens with esprima by using
npm run esprima-compare!
Template string interpolation
Template strings are matched as single tokens, from the starting
` to the
`, including interpolations (whose tokens are not matched
Matching template string interpolations requires recursive balancing of
Division and regex literals collision
Consider this example:
var g = 9.82 var number = bar / 2/g var regex = / 2/g
A human can easily understand that in the
number line we’re dealing with
division, and in the
regex line we’re dealing with a regex literal. How come?
Because humans can look at the whole code to put the
/ characters in context.
jsTokens regex scans throught the above, it will see the following
at the end of both the
It is then impossible to know if that is a regex literal, or part of an expression dealing with division.
Here is a similar case:
foo /= 2/g foo(/= 2/g)
The first line divides the
foo variable with
2/g. The second line calls the
foo function with the regex literal
/= 2/g. Again, since
sees forwards, it cannot tell the two cases apart.
There are some cases where we can tell division and regex literals apart, though.
First off, we have the simple cases where there’s only one slash in the line:
var foo = 2/g foo /= 2
Regex literals cannot contain newlines, so the above cases are correctly identified as division. Things are only problematic when there are more than one non-comment slash in a single line.
Secondly, not every character is a valid regex flag.
var number = bar / 2/e
The above example is also correctly identified as division, because
e is not a
valid regex flag. I initially wanted to future-proof by allowing
(any letter) as flags, but it is not worth it since it increases the amount of
ambigous cases. So only the standard
u flags are
allowed. This means that the above example will be identified as division as
long as you don’t rename the
e variable to some permutation of
gmiyu 1 to 5
Lastly, we can look forward for information.
- If the token following what looks like a regex literal is not valid after a regex literal, but is valid in a division expression, then the regex literal is treated as division instead. For example, a flagless regex cannot be followed by a string, number or name, but all of those three can be the denominator of a division.
- Generally, if what looks like a regex literal is followed by an operator, the
regex literal is treated as division instead. This is because regexes are
seldomly used with operators (such as
==), but division could likely be part of such an expression.
Please consult the regex source and the test cases for precise information on when regex or division is matched (should you need to know). In short, you could sum it up as:
If the end of a statement looks like a regex literal (even if it isn’t), it will be treated as one. Otherwise it should work as expected (if you write sane code).