CONSUME inside OPTION can't be disambiguated

[k1w1]

I think I must be missing something about how CONSUME should be used. I have a grammar where newlines are significant. Imagine this rule:

property = this.RULE('property', () => {
    this.OPTION1(() => this.CONSUME1(Newline));
    this.CONSUME2(Text);
    this.OPTION2(() => this.CONSUME3(Newline));
    this.CONSUME4(Colon);
    this.OPTION3(() => this.CONSUME5(Newline));
    this.CONSUME6(Text);
    this.OPTION4(() => this.CONSUME7(Newline));
  });

And this input: a : \n b

The problem I have is that the CST generated from this input looks like this:

{
  Colon: [1]
  Newline: [1] // Array with one element.
  Text: [2]
}

If the input contains one of the optional newlines, then the CST only contains one newline - but there is no positional information to tell me where that Newline goes. I was expecting the Newline array to contain something more like [null, null, Token, null] so I could tell it was the third optional Newline that is present.

What am I missing?

[msujew]

Hey @k1w1,

What am I missing?

You might have a misunderstanding of what the CST exactly represents. As the concrete syntax tree, it only contains what concretely exists within the input text - and that would be exactly one Colon, one Newline and two Text tokens. Nothing beyond that.

In order to figure out which of the options was actually invoked, you could analyze the offsets of the tokens. Though I'm generally not a big fan of using pure CST parsers, as they lose too much semantic information for my purposes (we built Langium with Chevrotain). I usually build an EmbeddedActionsParser and create an AST from the parsed input:

property = this.RULE('property', () => {
    const newlines = [null, null, null, null];
    this.OPTION1(() => newlines[0] = this.CONSUME1(Newline));
    const firstText = this.CONSUME2(Text);
    this.OPTION2(() => newlines[1] = this.CONSUME3(Newline));
    this.CONSUME4(Colon);
    this.OPTION3(() => newlines[2] =  this.CONSUME5(Newline));
    const secondText = this.CONSUME6(Text);
    this.OPTION4(() => newlines[3] = this.CONSUME7(Newline));
    return {
      newlines,
      firstText,
      secondText
    };
});

[k1w1]

The fact that Chevrotain generates a CST is ideal for my use-case because I need to be able to regenerate the source input, including whitespace, after manipulating the tree.

Separately, in my grammar newlines are significant in some places, but not in this rule which is why I have them here optionally. However when generating the text again from the CST I need to know which newlines were present so I can put them back.

Your suggestion for how to solve is a good one. But I am confused why it is necessary since the whole point of CST versus AST is to avoid problems like this. In my example newlines don't seem so important, but I can easily imagine a grammar where there were multiple optional tokens and you must know which one was present.

[k1w1]

Unfortunately in version 10.0.0 there was a change:

A RULE implementation function in CstParser is now defined as returning void.

So it looks like the solution you suggested will no longer work.

[k1w1]

But a bit more stepping through the code made me realize that there was a solution all along. The LABEL option for CONSUME exactly solves my problem:

property = this.RULE('property', () => {
    this.OPTION1(() => this.CONSUME1(Newline, { LABEL: 'Newline1' }));
    this.CONSUME2(Text);
    this.OPTION2(() => this.CONSUME3(Newline, { LABEL: 'Newline2' }));
    this.CONSUME4(Colon);
    this.OPTION3(() => this.CONSUME5(Newline, { LABEL: 'Newline3' }));
    this.CONSUME6(Text);
    this.OPTION4(() => this.CONSUME7(Newline, { LABEL: 'Newline4' }));
  });