How to access hidden CST tokens from AST?

[enikao]

I'm trying to retrieve the complete input text for a specific AST node, including all hidden tokens.

My example is a properties file:

integerProp = 1
myString = "Hello, world!"

So if I accessed AST model.props[0].$cstNode.content, I'd expect to see 5 tokens: ID, WS, EQ, WS, INT. But the two WS tokens are omitted.
I also tried to poke around in the model.$document structure, to no avail.
They seem to exist in lexerResult.hidden, but not propagated further?

[msujew]

Hey @enikao,

unlike in Xtext, whitespace tokens aren't actually stored in the CST and simply discarded. All other hidden tokens can be accessed via the CST though. You can change that behavior by overriding the TokenBuilder and change the group of the white space token to 'hidden'.

[szarnekow]

that information can easily be used to infer whitespace information without requiring to store whitespace explicitly

It's a deliberate decision to ignore windows line endings/unix line endings during full serialization?

[cdietrich]

there is no serialization, is there?

[msujew]

there is no serialization, is there?

Right, there is no serialization yet.

[shutterfreak]

Hey @enikao,

unlike in Xtext, whitespace tokens aren't actually stored in the CST and simply discarded. All other hidden tokens can be accessed via the CST though. You can change that behavior by overriding the TokenBuilder and change the group of the white space token to 'hidden'.

I created a custom TokenBuilder that extends DefaultTokenBuilder with the purpose to give my custom Fomatter access to WS (whitespace) tokens which the DefaultTokenBuilder discards by default. However, when I copy buildTerminalToken() from DefaultTokenBuilder and use it in my custom TokenBuilder, then the following line does something weird:

tokenType.GROUP = isWhitespace(regex) ? Lexer.SKIPPED : 'hidden';

Instead of setting tokenType.GROUP to 'hidden', it is set to 'This marks a skipped Token pattern, this means each token identified by it willbe consumed and then thrown into oblivion, this can be used to for example to completely ignore whitespace.'.

All I can come up with, is to explicitly check for is to skip the RegExpUtils.isWhitespace(regex) check, essentially removing the entire check. Is that the way to go?

[msujew]

Instead of setting tokenType.GROUP to 'hidden', it is set to 'This marks a skipped Token pattern, this means each token identified by it willbe consumed and then thrown into oblivion, this can be used to for example to completely ignore whitespace.'.

Note that this is the string value inside of Lexer.SKIPPED. This sounds like the expected behavior. If you want to change it, go ahead and remove the ternary.