Consider Token Categories in Lookahead Functions Calculation

[bd82]

Token Categories should be taken into account when performing lookahead calculation.

For example:

            const A = createToken({ name: "A" })
            
            const B = createToken({
                                   name: "B",
                                   // `B` is a typeof `A`
                                   categories: A 
                             })
            const C = createToken({ name: "C" })
            const D = createToken({ name: "D" })

            class CategoriesLookaheadBugParser extends Parser {
                constructor() {
                    super([A, B, C, D])
                    this.performSelfAnalysis()
                }

                public main = this.RULE("main", () => {
                    this.OR([
                        // 'alt1' is `BC`
                       // 'alt2' is `AD`
                      // However `B` can also match `A` due to the categories definition.
                     // So we need to look **two** tokens ahead
                    //  to decide between these two alternatives.
                        { ALT: () => this.SUBRULE(this.alt1) },
                        { ALT: () => this.SUBRULE(this.alt2) }
                    ])
                })

                public alt1 = this.RULE("alt1", () => {
                    this.CONSUME(B)
                    this.CONSUME(C)
                })

                public alt2 = this.RULE("alt2", () => {
                    this.CONSUME(A)
                    this.CONSUME(D)
                })
            }

[bd82]

This new capability will also allow to detect ambiguities that involve token categories.
e.g:

• https://sap.github.io/chevrotain/docs/guide/resolving_grammar_errors.html#COMMON_PREFIX

So while this change is more of a bug fix, it would be released in a new minor version as it could cause
previously "working" (incorrectly) grammars to fail during startup as previously missed errors would now be detected.