Rewriter needs to account for "dead code"

[retailcoder]

As of recently, all module-rewriting operations are carried through a TokenStreamRewriter. That's a huge leap forward and greatly improves undo-ability and overall module-rewriting accuracy, however there's a problem that wasn't addressed:

TokenStreamRewriter works off the lexer token stream, and that stream is "cleansed" so that the parser can work around a number of limitations:

• Line numbers are seen as WS tokens
• Precompiler blocks that aren't "live" are seen as WS tokens

Therefore, piping all module changes to TSR's has also introduced a nasty side-effect: we're turning the user's modules into how Rubberduck sees them. And that doesn't include line numbers and "dead" precompiler code paths.

We can't have all precompiler code paths in the token streams, and despite all our efforts anything that tries to get line numbers picked up by a parser rule breaks the grammar.

We can't release anything until we've found a solution to this problem - how do we go about it?

[retailcoder]

One possibility would be, as soon as we have the token stream for a module (well, after it's given to the parser for processing, I suppose), to get its rewriter and immediately replace the WS tokens we introduced with the line numbers and dead preprocessor code paths that we stripped from the module content string.

[Hosch250]

I'll take a crack at getting line numbers in the grammar. Can you provide different use cases I would need to account for?

[MDoerner]

Might we be able to solve the problem with the preprocessing directives by sending the parts to ignore to another channel instead of replacing them? Obviously, that would need some external variables in the lexer.

[retailcoder]

@MDoerner that's absolutely worth considering! ...except, wouldn't it mean we need embed the preprocessing logic into the lexer itself?

[comintern]

@Hosch250 - When I was looking at adding line numbers and labels to the grammar, I determined that it shouldn't be horribly difficult but it will probably require a bit of work. I think the key is to add something like an endOfLine token - basically any newline that doesn't have a line continuation. The main problem is that ANTLR uses greedy matching, so and lineNumber\label token would need to either back-reference to an endOfLine or be explicitly endOfLine (lineNumber | lineLabel). Like I said, shouldn't be difficult - just requires a sh!tload of adjustments to other contexts to make it work.

Re preprocessor directives, would it work if we just sent that token stream to a different ANTLR channel?

[MDoerner]

@retailcoder I think we can put them on another channel outside the lexer. The CommonToken has a settable field for the channel. So I think we can rewrite the channel during preprocessing.