flex-style named definitions cause ambiguity in re2c grammar

[skvadrik]

Bison reports 10 shift/reduce conflicts when compiling re2c parser. Turns out that all of them are caused by one unfortunate production in grammar:

decl -> FID expr

which stands for flex-style named definitions of the form:

name regular-expression

re2c tries to partially support flex syntax with '-F' flag. Native re2c named definitions are of the form:

name = regular-expression ;

Another notable difference is that re2c allows newlines inside of regular expressions in named definitions, while flex doesn't.
Both re2c and flex have rules of the form:

regular-expression action

re2c syntax allows to mix named definitions with rules. With native re2c named definitions that's ok: they have an ending semicolon that allows to distinguish them from rules. However, flex-style named definitions don't have an ending semicolon (newline acts as a delimiter in flex, but not in re2c), so mixing them with rules introduces parsing ambiguity. Consider the following example:

/*!re2c
    name "a"
    "b" "c" {}
*/

One can interpret this fragment in two different ways:

definition -> name "a" 
rule -> "b" "c" {}

and

definition -> name "a" "b"
rule -> "c" {}

both ways are valid, so there's a real ambiguity in grammar, not just some stupid LALR(1) conflict.

In flex, there's no parsing problem: it has newline as a delimiter and doesn't allow to mix named definitions with rules. Named definitions must all come together in a separate section delimited by "%%" :

definitions
%%
rules

As of now, re2c will fail to parse the example above. However, parsing problem vanishes in '-c' mode, because with '-c' rules have different form:

condition regular-expression action

Some re2c users (and notably, PHP team) use '-F' together with '-c' and don't face the parsing problem.

So what should we do? I see the following options:

1. drop flex-style syntax and -F option completely
   • breaks backwards compatibility for all -F users (users have to rewrite code to use re2c native named definitions), but at least the new code is compatible with older versions of re2c
   • easy to implement, re2c parser will become mush tidier
2. allow flex-style named definitions only in a separate section delimited by "%%", as flex does
   • breaks backwards compatibility for all -F users (users have to add "%%" and group named definitions together), the new code is incompatible with older re2c versions
   • not hard to implement
3. forbid newlines everywhere in regular expressions (both in named definitions and in rules)
   • breaks backwards compatibility in a nasty way for some unfortunate users
   • moderately hard to implement
4. forbid newlines only in flex-style named definitions, demand an ending newline in flex-style named definitions:
   • still may break some code, though conforming to flex syntax
   • very hard to implement, re2c parser will bloat immensely (I tried it and I really wouldn't recommend it)

I vote for (1) for the following reasons:

• Painful as it is, it is better than making new code incompatible with old re2c (2), breaking old code in obscure ways (3) or making re2c unmaintainable (4).
• I think that flex-style syntax is not very popular among re2c users. One notable exception is PHP team, but you folks are rapidly developing and responsive, and some syntax de-sugaring must be an easy job for you (or I can send a patch if you're short of time) :) You won't have to care for re2c version, both old and new will do.
• I think that two different syntaxes will never play well together and trying to merge them is a silly thing to do.

If (1) raises no objections, what should we do with -F option (remove or leave deprecated)?

Opinions welcome.

[skvadrik]

Fixed by forbidding newlines in flex-style named definitions. Turned out to be quite easy due to a hack in lexer.

e02b761#diff-9351147abfd7dc33744eaa3f4f30f371R199