Marker name is a string, not a lexer token

[dimbleby]

Discovered incidentally while working on python-poetry/poetry#5156 - sometimes a marker name was set to a lexer token rather than a string which, for instance, defeats comparisons for equality.

[abn]

Can we add some regression coverage for this one?

[dimbleby]

I must admit that I'm struggling now to recall why this was a problem. Per https://lark-parser.readthedocs.io/en/latest/classes.html#lark.Token

The Token class inherits from Python’s str, so normal string comparisons and operations will work as expected.

which seems like it should be fine, and my attempts to demonstrate otherwise in unit test are failing.

I'd be inclined to merge this anyway - it can't possibly be wrong, can it? But I am unable to prove that it is necessary, so if you're feeling super-cautious I guess don't.

Maybe we sometimes end up with different tokens for the same thing? This looks like an accident, I wonder whether that could confuse things sometimes?!?

[dimbleby]

I'm back on the trail... depending on where the parser is called from, you can end up with (eg) either Token(markers__MARKER_NAME, 'extra') or Token(MARKER_NAME, 'extra'); and these do not compare equal because they are not the same type of token

[dimbleby]

testcase added

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.5% Duplication

[finswimmer]

LGTM 👍

testcase added