Trailing . removed from token

[mattmasson]

Input text:

Access.

When I retrieve the lexer state, I'd expect a single token containing Access., but it looks like the trailing . is removed. The resulting LineTokens are equal to:

[{
    "kind": "Identifier",
    "positionStart": 0,
    "positionEnd": 6,
    "data": "Access"
}]

[JordanBoltonMN]

Take a look at the formal spec for identifier:

identifier:
    regular-identifier
    quoted-identifier

regular-identifier:
    available-identifier
    available-identifier dot-character regular-identifier

In short an identifier can't end with a period. The lexer reads available-identifier, then optionally reads . followed by available-identifier.

[mattmasson]

Ahhh - this likely explains why the Grammar treats Access.Database as one token, and Access. as two tokens. That creates an annoyance for intellisense purposes, though. I can't think of a case where you'd want . to be treated as a separate word/token.

I could fix this on the grammar side by treating . as a valid identifier char, but I was hoping to always rely on the positions returned by the parser code. (It would also break once VS Code allows us to stop using the grammar entirely).

Any suggestions on how to resolve?

I guess we could solve this within the intellisense code itself... if we find the last char is ., we adjust the current word position to include the previous identifier as well.

[JordanBoltonMN]

As far as I know there's no situation where an identifier followed by a dot isn't intended to be an identifier. I see two main options:

• Handle it in the Intellisense code
• Add an optional flag on how to tokenize identifiers (eg. Strict vs Intellisense)

Adding an additional flag would more tightly couple the parser's code with the VSCode extension, which is something I'd like to avoid if possible. That said if it's not easy to implement the fix in the Intellisense code and/or it requires a hacky solution then I'd prefer a flag over code separation ideology.