No generator, no whitespace? Please help

[docfate111]

How do I generate a ? None of the tests have whitespace and the input often gives invalid characters.
I got the g4 for https://github.com/antlr/grammars-v4/blob/master/javascript/javascript/

grammarinator-process JavaScriptParser.g4 JavaScriptLexer.g4 -o out --no-actions

grammarinator-generate -p out/JavaScriptUnparser.py -l out/JavaScriptUnlexer.py -r program -d 30 -n 3000

MacOS Python3.9
What is a generator do I need to write a generator program?
How do I write a transformer or change the grammar to add spaces? (Looking at the closed issue this is the solution but I have no idea what a transformer is).

[renatahodovan]

Hey @docfate111 !

The way you generate the generator is correct. With the command above the generated fuzzer, aka. generator will be placed in the out folder of your current working directory.

The way you use the grammarinator-generate utility is outdated: instead of referencing the generator file, you must use fully qualified names. In this case the command should look like:

grammarinator-generate JavaScriptGenerator.JavaScriptGenerator -r program -o tests/%d.js \
    -d 30 -n 3000 \
    --sys-path out/

The --sys-path argument ensures that the out directory is placed on the sys-path, hence referencing the generator with fully qualified name will work. Another example of the usage is available from the readme.

Your next problem will be that the original grammar specifies a superclass for both the lexer and the parser. These superclasses give additional information to the parser, which are useless for the fuzzer, ofc. Hence, I suggest to comment out these lines for now. If you still need some superclasses for your generated fuzzer, then you should write them for yourself.

To add spaces to your output you need to write a serializer. Serializers and transformers are similar things, they both work on grammar trees, but while serializers produce string outputs, transformers produce tree outputs. So, to add spaces to your output you need a serializer. A basic serializer, that inserts spaces after every UnparserRule (i.e., inner node of the grammar tree) is implemented here. To use it you should extend the previous generator command as shown below (serializers are also referred with fully qualified names just like the generator):

grammarinator-generate JavaScriptGenerator.JavaScriptGenerator -r program -o tests/%d.js \
    --serializer=grammarinator.runtime.serializer.simple_space_serializer \
    -d 30 -n 3000 \
    --sys-path out/

[akosthekiss]

@docfate111 I hope serializers (plus the additional details in @renatahodovan 's reply) did solve your problem. Closing this issue now as 5 months have passed since the Q and the A. However, if the problem still stands, feel free to re-open this issue (or open a new one).