keywords in terms

[AionNext]

I have defined In keyword "In" with the following pattern
createToken({ name: "In", pattern: /in/ })

when parsing terms that start with a prefix in (ie, inState) , the parser parses in as a keyword.

The solution is to define the pattern with a space following the keyword, like so
createToken({ name: "In", pattern: /in / })

However, it creates some other issues.
Is there any other way to fix this problem?

Thanks.

[msujew]

Looks like you ran into the typical keywords vs identifiers problem. See the documentation or the code example here on how to deal with it. It all comes down to the order of your token types in the lexer definition.

[msujew]

In practice you need to provide the { ..., longerAlt: ID } part for every token type which could be an ID if it weren't a keyword. For example:

createToken({ name: "Await", pattern: /await /, longerAlt: ID })
createToken({ name: "Define", pattern: /define/, longerAlt: ID })
createToken({ name: "Relation", pattern: /relation /, longerAlt: ID })
createToken({ name: "Between", pattern: /between /, longerAlt: ID })

However, with the way your system is set up, using longerAlt becomes quite difficult. You need the actual token type instance to use the longerAlt property. You need to define your ID token type before everything else and then add it to the token array (since the order is still important).

Note that I wouldn't recommend including whitespace in your keywords (such as the Await token). This will lead to lexing issues down the line.

[AionNext]

Hello,

I have moved ID definition ahead and now I am getting this error
lexer.zip

C:\xfer\parser\bizcod\parser>node .
C:\xfer\parser\bizcod\lib\chevrotain.js:1834
throw new Error("Errors detected in definition of Lexer:\n" +
^

Error: Errors detected in definition of Lexer:
Token: ->Await<- can never be matched.
Because it appears AFTER the Token Type ->ID<-in the lexer's definition.
See https://sap.github.io/chevrotain/docs/guide/resolving_lexer_errors.html#UNREACHABLE-----------------------
Token: ->Define<- can never be matched.
Because it appears AFTER the Token Type ->ID<-in the lexer's definition.
See https://sap.github.io/chevrotain/docs/guide/resolving_lexer_errors.html#UNREACHABLE-----------------------
Token: ->Relation<- can never be matched.

AS per your comment,
However, with the way your system is set up, using longerAlt becomes quite difficult.

Perhaps you can point me to an example which is properly set up so I can use it as a model?

Thanks

[msujew]

Like I said, you need to create the token first, but add it the token array in a specific fashion, i.e. as one of the last tokens:

const ID = { name: "ID", ... }
createToken({ name: "Await", pattern: /await /, longerAlt: ID })
createToken({ name: "Define", pattern: /define/, longerAlt: ID })
createToken({ name: "Relation", pattern: /relation /, longerAlt: ID })
createToken({ name: "Between", pattern: /between /, longerAlt: ID })
tokens.push(ID);

[AionNext]

Hello,

I fixes the previous problem but the original one still remains. I am attaching the modified file

lexer.zip

Also, I have removed some unused keywords so it becomes smaller.

I am not sure I have managed to handle this ID token properly

Thanks

[msujew]

I am not sure I have managed to handle this ID token properly

You haven't. You need to call the original createToken function on the ID terminal first.

const ID = orgCreateToken({ name: "ID", ... })
createToken({ name: "Await", pattern: /await /, longerAlt: ID })
createToken({ name: "Define", pattern: /define/, longerAlt: ID })
createToken({ name: "Relation", pattern: /relation /, longerAlt: ID })
createToken({ name: "Between", pattern: /between /, longerAlt: ID })
tokens.push(ID);