ANTLR backend: Spurious escaping of single quotes in character sets

[kaby76]

I'm using BNF.cf and bnfc 2.8.4 to generate an Antlr4 grammar which I then strip and use for my extension in a language server. The generated lexer code is syntactically incorrect and not accepted by the Antlr 4.9 tool.

STRINGTEXT : ~[\"\\] -> more;
CHARANY     :  ~[\'\\] -> more, mode(CHAREND);

I'm not sure why this isn't accepted by the Antlr grammar, after all, EscAny is in Antlr4's lexer rule. I will investigate further. But I recommend removing the backslashes before the single and double-quotes.

Disregard if this is already fixed.

[andreasabel]

Thanks @kaby76 for the report!
Please make sure that isn't already fixed in #319, and released with BNFC 2.9.0.

Should this not be fixed, please provide a MWE in form of a small .cf file that allow me to easily reproduce the bug.

[kaby76]

It is not fixed in v2.9.0, at least not completely. STRINGTEXT : ~["\\] -> more; is fine, but CHARANY : ~[\'\\] -> more, mode(CHAREND); is not.

Here is how to reproduce it from a Bash shell. I assume that you have Git, downloaded antlr-4.9-complete.jar (see https://www.antlr.org/download/index.html), downloaded Java (I have jdk-11.0.4), and placed the Java executable on path for your OS.

mkdir temp
cd temp
git clone https://github.com/BNFC/bnfc.git --branch v2.9.0 --single-branch
cd bnfc/source/src
cp ../BNFC.cf .
# I assume you are using bnfc 2.9.0 executable--I'm using that for Windows.
bnfc --java-antlr --force BNFC.cf
cd bnfc
# I assume you have downloaded the latest Antlr jar and java on path.
java -jar ~/Downloads/antlr-4.9-complete.jar *.g4

The output of this is:

warning(156): bnfcLexer.g4:78:16: invalid escape sequence \'
Exception in thread "main" java.lang.RuntimeException: set is empty
    at org.antlr.v4.runtime.misc.IntervalSet.getMaxElement(IntervalSet.java:421)
    at org.antlr.v4.runtime.atn.ATNSerializer.serialize(ATNSerializer.java:169)
    at org.antlr.v4.runtime.atn.ATNSerializer.getSerialized(ATNSerializer.java:601)
    at org.antlr.v4.Tool.generateInterpreterData(Tool.java:745)
    at org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:400)
    at org.antlr.v4.Tool.process(Tool.java:369)
    at org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:328)
    at org.antlr.v4.Tool.main(Tool.java:172)

[andreasabel]

Ok, this seems to be a problem with more recent ANTLR versions, it is still ok with 4.5:

$ java  org.antlr.v4.Tool 
ANTLR Parser Generator  Version 4.5.1
...
$ java  org.antlr.v4.Tool -lib bnfc -package bnfc bnfc/bnfcLexer.g4
$ java  org.antlr.v4.Tool -lib bnfc -package bnfc bnfc/bnfcParser.g4

[kaby76]

Ok, thanks. That kind of sucks. I'll check out Antlr's source and see what got messed up.

[andreasabel]

It seems like BNFC is still not implementing the escaping correctly, see the rules at: https://github.com/antlr/antlr4/blob/master/doc/lexer-rules.md
According to these rules, a single quote should not be escaped inside brackets (char sets).

CHARANY : ~[\'\\] -> more, mode(CHAREND);

This is wrong.

I suppose ANTLR got stricter checking the rules.

[kaby76]

I think it's a regression in the Antlr rules. This stuff was moved into lexer modes, which broke things. My first test using EscAny without modes works fine with [\'foobar], but I'm adding in the rules one by one to figure what broke the grammar. In any case, you might want to generate the single-quote without a backslash.

[kaby76]

There are at least two Git issues raised and then closed on escaped characters in Antlr: antlr/antlr4#1537 antlr/antlr4#1871 . For [\'], the tool gives a warning that it's not valid, but it crashes, which it shouldn't do. It is, apparently, now an invalid escape character, at least in character sets (https://github.com/antlr/antlr4/blob/master/doc/lexer-rules.md#lexer-rule-elements).