Data type using one terminal behaves differently from that terminal

[msujew]

Discussed in #1357

^{Originally posted by drhagen January 25, 2024}
If I have this grammar

grammar HelloWorld

entry Model:
    ((persons+=Person) EOL)*;

Person:
    'person' name=ID;

hidden terminal WS: /[ \t]+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal NEWLINE: '\n';
EOL returns string:
    NEWLINE;

and try to parse this file

person John
person Jane

I get an error "Expecting end of file but found 'person'".

If I replace EOL with NEWLINE and delete EOL entirely

grammar HelloWorld

entry Model:
    ((persons+=Person) NEWLINE)*;

Person:
    'person' name=ID;

hidden terminal WS: /[ \t]+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal NEWLINE: '\n';

the file parses successfully. What am I missing something about Langium because I would have expected those to be equivalent?

[msujew]

Seems like we only ever assume that data type rules call other data type rules in an unassigned way, see:

langium/packages/langium/src/parser/langium-parser.ts

Lines 236 to 251 in 0239db7

	// If we call a subrule without an assignment
	// We either append the result of the subrule (data type rule)
	// Or override the current object with the newly parsed object
	const current = this.current;
	if (isDataTypeNode(current)) {
	current.value += result.toString();
	} else {
	const resultKind = result.$type;
	const object = this.assignWithoutOverride(result, current);
	if (resultKind) {
	object.$type = resultKind;
	}
	const newItem = object;
	this.stack.pop();
	this.stack.push(newItem);
	}

The assumption breaks as soon as we call the data type rule from a normal parser rule without an assignment. I assume the solution would be to just ignore the result and store the CST created in the current AST node.