[RFC] Case Insensitivity Proof of Concept

[KvanTTT]

At present time there are quite a lot of case insensitive languages: Pascal, PHP, SQLite, TSQL and other. In the usual case this feature implemented via fragment rules (code generation approach, see here for example) or via overriding LA method in the input stream (runtime approach as mention @jimidle here).

I implemented new option and syntax in ANTLR to support case insensitivity. In my approach it is possible to declare caseInsensitive and caseSensitive modes (it can be used in grammar for parsing both PHP code (caseInsensitive keywords) and JavaScript code (caseSensitive keywords):

mode JavaScript, caseSensitive;
Break: 'break';
Do: 'do';
Instanceof: 'instanceof';
...

mode PHP, caseInsensitive;
Abstract: 'abstract';
Array: 'array';
As: 'as';

Also it is possible to declare caseInsensitive option for entire grammar (combined grammar also supported):

grammar pascal;
options { caseInsensitive = true; }
program
   : programHeading 'interface'? block '.'
   ;
programHeading
   : 'program' identifier ('(' identifierList ')')? ';'
   | 'unit' identifier ';'
   ;
// Another parser rules
...

INTERFACE: 'interface';
PROGRAM: 'program';
UNIT: 'unit';
// Another tokens with string literals (without fragment rules).

Moreover it is possible to declare caseSensitive and caseInsensitive tokens in the same mode like this:

lexer grammar L;
options { caseInsensitive = true; }
Token_1: 'a';

mode DEFAULT_MODE, caseSensitive;
Token_3: 'd'

See unit tests for detail and other cases.

Сase insensitivity implemented via code generation approach, but it can be replaced with runtime approach too if necessary. This feature improve grammar readability and make grammar creation more easy.

If this pull request overall is good, I'll finish remaining issues with separated lexer and parser and other. Otherwise I'll try to separate fixed symbol issues and make another pull request. Suggestions are welcome.

[KvanTTT]

Sorry, I did not run runtime tests. I'll fix it.

[KvanTTT]

I have not idea why "Method L:_serializeATN () is too complex" exception now occured for C# runtime. 😒

[ericvergnaud]

Probably because the recursive string concatenation ("abc" + "bcd" + … ) is now exceeding the compiler stack limit

Le 9 janv. 2016 à 07:00, Ivan Kochurkin notifications@github.com a écrit :

I have not idea why "Method L:_serializeATN () is too complex" exception occured for C# runtime.

—
Reply to this email directly or view it on GitHub #1092 (comment).

[KvanTTT]

@ericvergnaud, no, this issue related to lexer. Maybe method has too big size. Local tests are passed (Windows 7). I tried to disable "inline" optimization as described here, but test still failed.
I don't understangd why it occured after my pull request.

[ericvergnaud]

lexer also has its own serializedATN, and the failing method is TestHugeLexer
I haven’t look at your PR, but given the nature of the proposal, I guess it almost doubles the size of the ATN.

Le 10 janv. 2016 à 19:12, Ivan Kochurkin notifications@github.com a écrit :

@ericvergnaud https://github.com/ericvergnaud, no, this issue related to lexer. Maybe method has too big size. Local tests are passed (Windows 7). I tried to disable "inline" optimization as described here https://bugzilla.xamarin.com/show_bug.cgi?id=5092, but test still failed.
I don't understangd why it occured after my pull request.

—
Reply to this email directly or view it on GitHub #1092 (comment).

[KvanTTT]

@ericvergnaud, I understand you, but in this test serializedATN size should not be changed because of caseInsensitive option is not enabled. But I'll look at it in more detail.

[parrt]

Any change to the meta-language requires deep thought on my part. Not sure when I can devote the time.

[parrt]

Hi. I'm going to close not because it's not an excellent job but because it's a fairly significant change and I'm nervous about unintended consequences.