Simplify output path generation; new -Xexact-output-dir option

[mike-lischke]

This is a new version of PR #1829 from 19 April and #1905 (which lost all the changes I made during a git merge).

The twisted way to create a javac like output path breaks many use cases, including very simple ones with lexer + parser grammar in a relative subdir (where parser generation doesn't find the lexer tokens). Additionally, not all projects prefer to have their generated files in an output folder which is comprised of the given output path and the subdir given for the grammar files. The check for absolute paths as decision criterion is a weak one anyway - it makes simple usage difficult and doesn't deal with path expansion on all platforms.

If a javac like output path is required it should be easy to construct this upfront and provide it as the output path parameter, instead of implicitely creating it. This patch simplifies things without any WTF moments because generated files end up at unexpected locations. It now can deal with simple cases (no output, no grammar subdir) up to complicated cases (nested and/or absolute output dir, multiple grammar subfolders).

If more than one output folder is wanted run the tool multiple times with different output path settings and add the lib option if you need to reference previously created files.

It might look like a disadvantage that now generation of parser files ends up at a different path when run with relative grammar subfolders, but actually that was broken anyway so I don't expect anyone having used that (but instead either absolute paths or no subfolders at all).

Fixes #1087, #753, #638, tunnelvisionlabs/antlr4ts#306, tunnelvisionlabs/antlr4ts#303

[parrt]

@mike-lischke I'm working on integrating this one now. I will need to add an option so that I can leave everything as backward-compatible. trying to understand the Delta and the functionality from your changes and based on our discussions above. hang on...

[parrt]

If I may summarize the changes...

1. Output -o directory specifier is the exact directory containing the output. Previously it would include the relative path specified on the grammar itself for the purposes of packages.

new: -o /tmp subdir/T.g4 => /tmp/subdir/T.java
old: -o /tmp subdir/T.g4 => /tmp/T.java

2. Previously we looked for the tokens vocab file in the -lib dir or in the output dir. New: also look in the directory containing the grammar, particularly if it it is specified with a path.

Example for the output directory (4.7)

Here is the existing 4.7 functionality.

$ cd /tmp/parrt
$ tree
.
├── B.g4
└── src
    └── pkg
        └── A.g4
$ a4.7 -o /tmp/build src/pkg/A.g4
$ tree /tmp/build
/tmp/build/
└── src
    └── pkg
        ├── A.tokens
        ├── ABaseListener.java
        ├── ALexer.java
        ├── ALexer.tokens
        ├── AListener.java
        └── AParser.java

Now, let's build a grammar that sits in the current directory:

$ a4.7 -o /tmp/build B.g4
$ tree /tmp/build
/tmp/build
├── B.tokens
├── BBaseListener.java
├── BLexer.java
├── BLexer.tokens
├── BListener.java
├── BParser.java
└── src
    └── pkg
        ├── A.tokens
        ├── ABaseListener.java
        ├── ALexer.java
        ├── ALexer.tokens
        ├── AListener.java
        └── AParser.java

Finally, if we don't specify the output directory, it paid attention to the relative path specified on the input grammar:

$ a4.7 src/pkg/A.g4
$ tree
.
├── B.g4
└── src
    └── pkg
        ├── A.g4
        ├── A.tokens
        ├── ABaseListener.java
        ├── ALexer.java
        ├── ALexer.tokens
        ├── AListener.java
        └── AParser.java

Example for the output directory (4.7.1 with -Xexact-output-dir)

Now, the output directory is the exact directory where output is generated regardless of relative paths on the grammar

$ cd /tmp/parrt
$ a4.7.1 -Xexact-output-dir  -o /tmp/build src/pkg/A.g4
$ tree /tmp/build
/tmp/build
├── A.tokens
├── ABaseListener.java
├── ALexer.java
├── ALexer.tokens
├── AListener.java
└── AParser.java

If you use the package option, it still does not change where the output is generated if you use -o

$ a4.7.1 -Xexact-output-dir -package pkg -o /tmp/build src/pkg/A.g4
$ tree /tmp/build
/tmp/build
├── A.tokens
├── ABaseListener.java
├── ALexer.java
├── ALexer.tokens
├── AListener.java
└── AParser.java

4.7.1 does however add the package specification into the generated files:

$ grep package /tmp/build/A*.java
/tmp/build/ABaseListener.java:package pkg;
/tmp/build/ALexer.java:package pkg;
/tmp/build/AListener.java:package pkg;
/tmp/build/AParser.java:package pkg;

Compare this to 4.7:

$ a4.7 -package pkg -o /tmp/build src/pkg/A.g4
beast:/tmp/parrt $ tree /tmp/build
/tmp/build
└── src
    └── pkg
        ├── A.tokens
        ├── ABaseListener.java
        ├── ALexer.java
        ├── ALexer.tokens
        ├── AListener.java
        └── AParser.java

Example of where it looks for tokens vocab

In 4.7, we got an error for an obvious case that should work:

$ cd /tmp/parrt
$ tree
.
└── src
    └── pkg
        ├── L.g4
        └── P.g4
$ a4.7 -o /tmp/build src/pkg/*.g4
error(160): P.g4:2:21: cannot find tokens file /tmp/build/L.tokens
warning(125): P.g4:3:4: implicit definition of token A in parser

In 4.7.1 it looks in the directory containing the grammars as well:

$ a4.7.1 -o /tmp/build src/pkg/*.g4
$ tree /tmp/build
/tmp/build
├── L.java
├── L.tokens
├── P.java
├── P.tokens
├── PBaseListener.java
├── PListener.java
└── src
    └── pkg
        ├── L.java
        └── L.tokens

[parrt]

Merged manually after adding -Xexact-output-dir

[DamianReeves]

Is there a combination of -package and -output that works properly for 4.7? I'm stuck on 4.7 and it will be weeks before I can get 4.7.1 in my organization. I bump up with this on the regular and am feeling the pain.

My scenario looks like the one described above for 4.7:

$ cd /tmp/parrt
$ tree
.
└── src
    └── pkg
        ├── L.g4
        └── P.g4
$ a4.7 -o /tmp/build src/pkg/*.g4 -package "com.myorg.myproj"
error(160): P.g4:2:21: cannot find tokens file /tmp/build/L.tokens
warning(125): P.g4:3:4: implicit definition of token A in parser

[parrt]

Depends on "properly" I guess. Most people have had no trouble doing what they need beforehand so maybe it's a matter or reorg of output dirs?