Fix for offside errors on interpolated expressions

[abonie]

Fixes: #13728

The problem occurred when INTERP_STRING_END token was offside in relation to INTERP_STRING_BEGIN_PART token. Too many contexts would be taken off the offside stack in hwFetchToken in LexFilter which would result in unfinished block error.

It seems to me that in case of interpolated strings, we can completely skip closing surrounding contexts.

[abonie]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[abonie]

@auduchinok given that you have done a lot of work in the parser - I can't think of any case that this change would break and it seems to fix the original issue as expected. However, I have difficult time fully wrapping my head around how the contexts' stack is used by hwTokenFetch (and debugging this function is pretty much impossible). Do you have any idea/intuition regarding what regression this change could cause?

[auduchinok]

@auduchinok given that you have done a lot of work in the parser - I can't think of any case that this change would break and it seems to fix the original issue as expected. However, I have difficult time fully wrapping my head around how the contexts' stack is used by hwTokenFetch (and debugging this function is pretty much impossible). Do you have any idea/intuition regarding what regression this change could cause?

No, from the top of my head, I don't expect anything to break from these changes.
Just in case, using fsc with --tokenize-debug flag is a tolerable substitute for debugging hwTokenFetch, as it prints a lot of info about the offset stack.

[auduchinok]

@abonie Could you, maybe, add some parser tests, so we have a particular tree structure captured with these changes?

[abonie]

Neat, I didn't know about the --tokenize-debug and I was just manually setting the forceDebug to true.

Added some syntax tree tests.