Fix corner cases for the greedy flag

[zeitgeist87]

By refactoring the tokenize() method into two separate methods, it is
possible to recursively clean up any broken tokens left by the greedy
feature.

This should fix the issue #1075.

[zeitgeist87]

Actually this is not a good solution after all. It only works if the comment contains another comment.

[zeitgeist87]

The same error occurs with a template string in JS:

[Golmote]

Thank you for taking a look into this!
I think your idea should be able to fix even the template string example. After the part //" ` sdsf ` is invalidated, we could reprocess only the remaining fragment, that is ` sdsf `, which should highlight as a template string, right?

[zeitgeist87]

Yes I think it is possible to optimize and generalize this approach. I will try a few things and then update this PR.

[zeitgeist87]

I don't think we can make the greedy flag perfect. It was a compromise from the beginning and there will always be a few corner cases where it fails.

At some point we have to decide if the extra effort of fixing greedy is worth it.

[zeitgeist87]

It should be fixed now. I've also added test cases to a few languages including one test case for template strings in Javascript.

[zeitgeist87]

@Golmote Do you think this can be merged?

[Golmote]

@zeitgeist87 Again sorry for the long time before review. This looks good to me. Thank you so much for digging into this.

Feel free to merge when you can!

[zeitgeist87]

@Golmote Thanks for the review! I've rebased it and solved the merge conflicts.