Recovery Re-Sync logic ignores Token Categories

[bd82]

It seems the tokenMatcher function is not used which means
the token categories are not handled properly.

Example:

• https://github.com/SAP/chevrotain/blob/c001c1a8f94f0d00de03a10be701bcb28a5e34be/packages/chevrotain/src/parse/parser/traits/recoverable.ts#L278-L291

Possibly also affects this:

• https://github.com/SAP/chevrotain/blob/c001c1a8f94f0d00de03a10be701bcb28a5e34be/packages/chevrotain/src/parse/parser/traits/recoverable.ts#L269-L276

This may cause the Parser to re-sync (throw out) more input than needed to recover.

[AlansCodeLog]

Edit: Nevermind, I think the problem was something else and also maybe I misunderstood how the recovery is supposed to work*, but, I got things working again and recovering how I wanted.

Just ran into this. I'm writing a parser that takes both symbol and non-symbol operators, so they are distinct token types (since the word operators must use longer_alt but not the symbol ones), but the operators have a category to be able to consume them since MANY_SEP won't take rules, unfortunately using it leads to this issue.

The following workaround works but is very wordy since DEF must be written twice.

// $.MANY_SEP({ SEP: OPERATOR_AND, DEF })
$.OR([
	{
		GATE: () => $.LA(2).tokenType === SYM_AND,
		ALT: () => $.MANY_SEP({ SEP: SYM_AND, DEF }),
	},
	{
		GATE: () => $.LA(2).tokenType !== SYM_AND,
		ALT: () => $.MANY_SEP2({ SEP: WORD_AND, DEF2 })
	},
])

Is there any danger in just using js control flows like this if it seems to work?

if ($.LA(2).tokenType === SYM_AND) {
	$.MANY_SEP({ SEP: SYM_AND, DEF })
} else {
	$.MANY_SEP({ SEP: WORD_AND, DEF })
}

[medihack]

I am writing an XML-like parser where I need some recovery for something like < <Foo />. I had a hard time, as I could not get the re-sync to work. Then I found a hint to this issue in the xml-tools repository. It seems I could fix this with overwriting findReSyncTokenType, too. But it feels more like a hack as it is not even a public API (no type definitions). From the source code of xml-tools it seems very easy to fix it upstream in Chevrotain. Is there a reason why it is not fixed yet?
I also wonder a bit if more of those recovery APIs should be public and with type definitions (especially as I already hat the problem with #1753).

[bd82]

it seems very easy to fix it upstream in Chevrotain. Is there a reason why it is not fixed yet?

Priorities / not affecting my use cases so much / leaving some easy to fix items for contributors / ...

I also wonder a bit if more of those recovery APIs should be public and with type definitions

Yeah, that sounds like an improvement for one of the big (and somewhat unique) features of fault tolerance.

[medihack]

Chevrotain is really a nice project and makes my life so much easier as I create a quite complex medical template language (planned to be released later this year under an Open Source license). I also create the complete tooling around it (language service with Monaco and VS Code integration). So along the way, I will see where I can contribute.

[bd82]

I will see where I can contribute.

Awesome 👍