Skip to content

Commit

Permalink
docs: fix typos, punctuation in keywords_vs_identifiers.js (#1945)
Browse files Browse the repository at this point in the history
Fix typos and punctuation, and improve some wordings a little.
  • Loading branch information
srfsh committed Jun 20, 2023
1 parent 9841b81 commit d3be71e
Showing 1 changed file with 11 additions and 11 deletions.
22 changes: 11 additions & 11 deletions examples/lexer/keywords_vs_identifiers/keywords_vs_identifiers.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@
* This example shows how to resolve the keywords vs identifiers ambiguity.
* The order of TokenTypes in the lexer definition matters.
*
* If we place keywords before the Identifier than identifiers that have a keyword like prefix will not be
* lexer correctly. For example, the keyword "for" and the identifier "formal".
* If we place keywords before the Identifier, then identifiers that have a keyword-like prefix will not be
* lexed correctly. For example, the keyword "for" and the identifier "formal".
* The token vector will be ["for", "mal"] instead of ["formal"].
*
* On the other hand if we place keywords after the Identifier than they will never
* be lexed as keywords as all keywords are usually valid identifiers.
* On the other hand, if we place keywords after the Identifier, then they will never
* be lexed as keywords, as all keywords are usually valid identifiers.
*
* The solution is to place the keywords BEFORE the Identifier with an added property
* The solution is to place the keywords BEFORE the Identifier with an additional property, longer_alt,
* telling the lexer to prefer the longer identifier alternative if one is found.
*/

Expand Down Expand Up @@ -40,14 +40,14 @@ const Whitespace = createToken({
})

const keywordsVsIdentifiersLexer = new Lexer([
Whitespace, // Whitespace is very common in most languages so placing it first generally speeds up the lexing.
Whitespace, // Whitespace is very common in most languages so placing it first generally speeds up the lexing process.

While, // the actual keywords (While/For/Do) must appear BEFORE the Identifier Token as they are all a valid prefix of it's PATTERN.
For, // However the edge case of an Identifier with a prefix which is a valid keyword must still be handled, for example:
Do, // 'do' vs 'done' or 'for' vs 'forEach'. This is solved by defining 'Keyword.LONGER_ALT = Identifier'/
// thus each time a Keyword is detected the Lexer will also try to match a LONGER Identifier..
While, // the actual keywords (While/For/Do) must appear BEFORE the Identifier Token, as they are all a valid prefix of it's PATTERN.
For, // However, the edge case of an Identifier with a prefix, which is a valid keyword, must still be handled. For example:
Do, // 'do' vs 'done' or 'for' vs 'forEach'. This is solved by defining 'Keyword.LONGER_ALT = Identifier',
// thus each time a Keyword is detected, the Lexer will also try to match a LONGER Identifier.

Identifier // As mentioned above, the Identifier Token must appear after ALL the Keyword Tokens
Identifier // As mentioned above, the Identifier Token must appear after ALL the Keyword Tokens.
])

module.exports = {
Expand Down

0 comments on commit d3be71e

Please sign in to comment.