weird loc tracking

[KFlash]

@3cp See this test? https://github.com/meriyah/meriyah/blob/master/test/lexer/whitespace.ts#L228

Single line comment shouldn't have any LT - it should break. So shouldn't this test simply have zero LT and hasNext set to true?
Next time you invoke the nextToken it should consume the line terminator.

There are other similar tests too.

https://github.com/meriyah/meriyah/blob/master/test/lexer/whitespace.ts#L641

Here shouldn't the index stop at 5 and not 8?

cc / @fisker What do you think?

[3cp]

That sounds right. I knew meriyah always put leading space into next token.

What parser issue did this quirk contribute to?

[KFlash]

I just compared Meriyah output against my private code when I noticed this. I don't know about any real world cases except maybe for Prettier?
It also makes it difficult to extract the comment right if you want to extract them by given pos.

[KFlash]

@3cp ideas for an fix?

[3cp]

The test tests scanned token by skipping comments. It looks fine. I have cleaned up the whitespace unit tests file for some missing check.