@@ -207,7 +207,7 @@ Rule elements specify what the parser should do at a given moment just like stat
Match token T at the current input position. Tokens always begin with a capital letter.
-
’literal’
+
'literal'
Match the string literal at the current input position. A string literal is simply a token with a fixed string.
@@ -232,7 +232,7 @@ Match any single token except for the end of file token. The “dot” operator
-When you want to match everything but a particular token or set of tokens, use the `~` “not” operator. This operator is rarely used in the parser but is available. `~INT` matches any token except the `INT` token. `~’,’` matches any token except the comma. `~(INT|ID)` matches any token except an INT or an ID.
+When you want to match everything but a particular token or set of tokens, use the `~` “not” operator. This operator is rarely used in the parser but is available. `~INT` matches any token except the `INT` token. `~','` matches any token except the comma. `~(INT|ID)` matches any token except an INT or an ID.
Token, string literal, and semantic predicate rule elements can take options. See Rule Element Options.
@@ -486,4 +486,4 @@ Invalid input would cause `config` to return immediately without matching any in
```
file : element* EOF; // don't stop early. must match all input
-```
\ No newline at end of file
+```
diff --git a/doc/php-target.md b/doc/php-target.md
index 75eae465ce..a87410fff9 100644
--- a/doc/php-target.md
+++ b/doc/php-target.md
@@ -15,7 +15,7 @@ generated by ANTLR4. The runtime provides a common set of tools for using your p
Install the runtime with Composer:
```bash
-composer install antlr/antlr4
+composer require antlr/antlr4-php-runtime
```
#### 3. Generate your parser
@@ -84,7 +84,6 @@ $lexer = new JSONLexer($input);
$tokens = new CommonTokenStream($lexer);
$parser = new JSONParser($tokens);
$parser->addErrorListener(new DiagnosticErrorListener());
-$parser->setBuildParseTree(true);
$tree = $parser->json();
ParseTreeWalker::default()->walk(new TreeShapeListener(), $tree);
@@ -108,4 +107,4 @@ The expected output is:
{"a":1}
"a":1
1
-```
\ No newline at end of file
+```
diff --git a/doc/predicates.md b/doc/predicates.md
index 09998c4257..e04d2464b0 100644
--- a/doc/predicates.md
+++ b/doc/predicates.md
@@ -136,7 +136,7 @@ If, on the other hand, the next character after input `enum` is a letter, then o
Predicates come into play by pruning the set of viable lexer rules. When the lexer encounters a false predicate, it deactivates that rule just like parsers deactivate alternatives with false predicates.
-Like parser predicates, lexer predicates can't depend on side effects from lexer actions. That's because actions can only execute after the lexer positively identifies the rule to match. Since predicates are part of the rule selection process, they can't rely on action side effects. Lexer actions must appear after predicates in lexer rules. As an example, here's another way to match enum as a keyword in the lexer:
+Like parser predicates, lexer predicates can't depend on side effects from lexer actions. That said, the predicate can depend on a side effect of an action that occured during the recognition of the previous token. That's because actions can only execute after the lexer positively identifies the rule to match. Since predicates are part of the rule selection process, they can't rely on action side effects created by actions in currently-prospective rules. Lexer actions must appear after predicates in lexer rules. As an example, here's another way to match enum as a keyword in the lexer:
```
ENUM: [a-z]+ {getText().equals("enum")}?
@@ -162,3 +162,17 @@ That works great, but it's really just for instructional purposes. It's easier t
```
ENUM : 'enum' ;
```
+
+Here's another example of a predicate. It's important to note that the predicate is evaluated before the action because actions are only executed if the lexer rule matches. The actions are not executed in line; they are collected and executed en mass later.
+
+```
+INDENT : [ \t]+ {System.out.println("INDENT")>} {this.getCharPositionInLine()==0}? ;
+```
+
+For more information on how actions and predicates operate in the lexer, see [Lexer actions and semantic predicates are executed out of order](https://github.com/antlr/antlr4/issues/3611) and [Lexer.getCharIndex() return value not behaving as expected](https://github.com/antlr/antlr4/issues/3606). The lexer rule that will not work as expected is:
+
+```
+Stuff : ( 'a'+ {count++;} | 'b') 'c' 'd' {count == 3}? ;
+```
+
+The `count++` code we'll not execute until after `Stuff` has been recognized (assuming count!=3).
\ No newline at end of file
diff --git a/doc/python-target.md b/doc/python-target.md
index 7ed73c281e..1ef34e74d5 100644
--- a/doc/python-target.md
+++ b/doc/python-target.md
@@ -1,129 +1,262 @@
-# Python (2 and 3)
+# Python 3
-The examples from the ANTLR 4 book converted to Python are [here](https://github.com/jszheng/py3antlr4book).
+## Requirements
-There are 2 Python targets: `Python2` and `Python3`. This is because there is only limited compatibility between those 2 versions of the language. Please refer to the [Python documentation](https://wiki.python.org/moin/Python2orPython3) for full details.
-
-How to create a Python lexer or parser?
-This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example:
-
-```
-$ antlr4 -Dlanguage=Python2 MyGrammar.g4
-```
-
-or
-
-```
-$ antlr4 -Dlanguage=Python3 MyGrammar.g4
-```
-
-For a full list of antlr4 tool options, please visit the tool documentation page.
-
-## Where can I get the runtime?
-
-Once you've generated the lexer and/or parser code, you need to download the runtime. The Python runtimes are available from PyPI:
-
-* https://pypi.python.org/pypi/antlr4-python2-runtime/
-* https://pypi.python.org/pypi/antlr4-python3-runtime/
-
-The runtimes are provided in the form of source code, so no additional installation is required.
-
-We will not document here how to refer to the runtime from your Python project, since this would differ a lot depending on your project type and IDE.
-
-## How do I run the generated lexer and/or parser?
-
-Let's suppose that your grammar is named, as above, "MyGrammar". Let's suppose this parser comprises a rule named "startRule". The tool will have generated for you the following files:
-
-* MyGrammarLexer.py
-* MyGrammarParser.py
-* MyGrammarListener.py (if you have not activated the -no-listener option)
-* MyGrammarVisitor.py (if you have activated the -visitor option)
-
-(Developers used to Java/C# AntLR will notice that there is no base listener or visitor generated, this is because Python having no support for interfaces, the generated listener and visitor are fully fledged classes)
-
-Now a fully functioning script might look like the following:
-
-```python
-import sys
-from antlr4 import *
-from MyGrammarLexer import MyGrammarLexer
-from MyGrammarParser import MyGrammarParser
-
-def main(argv):
- input_stream = FileStream(argv[1])
- lexer = MyGrammarLexer(input_stream)
- stream = CommonTokenStream(lexer)
- parser = MyGrammarParser(stream)
- tree = parser.startRule()
-
-if __name__ == '__main__':
- main(sys.argv)
-```
-
-This program will work. But it won't be useful unless you do one of the following:
-
-* you visit the parse tree using a custom listener
-* you visit the parse tree using a custom visitor
-* your grammar comprises production code (like ANTLR3)
-
-(please note that production code is target specific, so you can't have multi target grammars that include production code, except for very limited use cases, see below)
-
-## How do I create and run a custom listener?
-
-Let's suppose your MyGrammar grammar comprises 2 rules: "key" and "value". The antlr4 tool will have generated the following listener:
-
-```python
-class MyGrammarListener(ParseTreeListener):
- def enterKey(self, ctx):
- pass
- def exitKey(self, ctx):
- pass
- def enterValue(self, ctx):
- pass
- def exitValue(self, ctx):
- pass
-```
-
-In order to provide custom behavior, you might want to create the following class:
-
-```python
-class KeyPrinter(MyGrammarListener):
- def exitKey(self, ctx):
- print("Oh, a key!")
-```
-
-In order to execute this listener, you would simply add the following lines to the above code:
-
-```
- ...
- tree = parser.startRule() - only repeated here for reference
- printer = KeyPrinter()
- walker = ParseTreeWalker()
- walker.walk(printer, tree)
-```
-
-Further information can be found from the ANTLR 4 definitive guide.
+You will need to install Python and Pip, version 3.6 or better.
+See https://www.python.org/downloads/
+and https://www.geeksforgeeks.org/how-to-install-pip-on-windows/.
+
+## A simple example targeting Python3
+
+An example of a parser for the Python3 target consists of the following files.
+* An Antlr4 grammar, e.g., Expr.g4:
+ ```antlr
+ grammar Expr;
+ start_ : expr (';' expr)* EOF;
+ expr : atom | ('+' | '-') expr | expr '**' expr | expr ('*' | '/') expr | expr ('+' | '-') expr | '(' expr ')' | atom ;
+ atom : INT ;
+ INT : [0-9]+ ;
+ WS : [ \t\n\r]+ -> skip ;
+ ```
+* Driver.py:
+The driver code opens a file, creates a lexer, token stream,
+and parser, then calls the parser.
+ ```python
+ import sys
+ from antlr4 import *
+ from ExprLexer import ExprLexer
+ from ExprParser import ExprParser
+ from VisitorInterp import VisitorInterp
+
+ def main(argv):
+ input_stream = FileStream(argv[1])
+ lexer = ExprLexer(input_stream)
+ stream = CommonTokenStream(lexer)
+ parser = ExprParser(stream)
+ tree = parser.start_()
+
+ if __name__ == '__main__':
+ main(sys.argv)
+ ```
+* requirements.txt:
+This file contains a list of the
+required packages for the program. Required
+packages are downloaded by `pip`. The file
+must include a reference to the Antlr Python3 runtime.
+ ```
+ antlr4-python3-runtime==4.13.0
+ ```
+* A build script, e.g., build.sh:
+You should provide a script that builds the program.
+ ```
+ pip install -r requirements.txt
+ antlr4 -Dlanguage=Python3 Expr.g4
+ ```
+_It is vital that the versions for the
+Antlr tool used to generate the parser
+and the Antlr Python3 runtime match.
+E.g., 4.13.0. Using build files will help
+eliminate common errors from happening._
+
+_For a list of antlr4 tool options, please visit [ANTLR Tool Command Line Options](https://github.com/antlr/antlr4/blob/master/doc/tool-options.md)._
+* Input, e.g., input.txt:
+ ```
+ -(1 + 2)/3;
+ 1;
+ 2+3;
+ 8*9
+ ```
+* A run script, which runs your program.
+ ```
+ python Driver.py input.txt
+ ```
+
+## Parse tree traversal
+
+Tree traversal is used to implement
+[static](https://en.wikipedia.org/wiki/Static_program_analysis) or [dynamic](https://en.wikipedia.org/wiki/Dynamic_program_analysis)
+program analysis.
+Antlr generates two types of tree traversals: visitors and listeners.
+
+Understanding when to choose a visitor versus a listener is a good idea.
+For further information, see https://tomassetti.me/listeners-and-visitors/.
+
+A visitor is the best choice when computing only a single [synthesized attribute](https://en.wikipedia.org/wiki/Attribute_grammar#Synthesized_attributes)
+or when you want to control the order of parse tree nodes visited.
+Alternatively, a listener is the best choice when computing both synthesized
+and [inherited attributes](https://en.wikipedia.org/wiki/Attribute_grammar#Inherited_attributes).
+
+In many situations, they are interchangeable.
+
+### Visitors
+
+Antlr visitors generally implement a post-order tree walk. If you write
+`visit...` methods, the method must contain code to visit the children
+in the order you want. For a post-order tree walk, visit the children first.
+
+To implement a visitor, add the `-visitor` option to the `antlr4` command.
+Create a class that inherits from the generated visitor,
+then add `visit` methods that implement the analysis. Your driver code
+should call the `visit()` method for the root of the parse tree.
+
+For example, the following code implements an expression evaluator for the Expr.g4 grammar using a visitor.
-The Python implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for Python.
+* Driver.py:
+ ```python
+ import sys
+ from antlr4 import *
+ from ExprLexer import ExprLexer
+ from ExprParser import ExprParser
+ from VisitorInterp import VisitorInterp
-## Target agnostic grammars
+ def main(argv):
+ input_stream = FileStream(argv[1])
+ lexer = ExprLexer(input_stream)
+ stream = CommonTokenStream(lexer)
+ parser = ExprParser(stream)
+ tree = parser.start_()
+ if parser.getNumberOfSyntaxErrors() > 0:
+ print("syntax errors")
+ else:
+ vinterp = VisitorInterp()
+ vinterp.visit(tree)
-If your grammar is targeted to Python only, you may ignore the following. But if your goal is to get your Java parser to also run in Python, then you might find it useful.
+ if __name__ == '__main__':
+ main(sys.argv)
+ ```
+* VisitorInterp.py:
+ ```python
+ import sys
+ from antlr4 import *
+ from ExprParser import ExprParser
+ from ExprVisitor import ExprVisitor
-1. Do not embed production code inside your grammar. This is not portable and will not be. Move all your code to listeners or visitors.
-1. The only production code absolutely required to sit with the grammar should be semantic predicates, like:
-```
-ID {$text.equals("test")}?
-```
+ class VisitorInterp(ExprVisitor):
+ def visitAtom(self, ctx:ExprParser.AtomContext):
+ return int(ctx.getText())
-Unfortunately, this is not portable, but you can work around it. The trick involves:
+ def visitExpr(self, ctx:ExprParser.ExprContext):
+ if ctx.getChildCount() == 3:
+ if ctx.getChild(0).getText() == "(":
+ return self.visit(ctx.getChild(1))
+ op = ctx.getChild(1).getText()
+ v1 = self.visit(ctx.getChild(0))
+ v2 = self.visit(ctx.getChild(2))
+ if op == "+":
+ return v1 + v2
+ if op == "-":
+ return v1 - v2
+ if op == "*":
+ return v1 * v2
+ if op == "/":
+ return v1 / v2
+ return 0
+ if ctx.getChildCount() == 2:
+ opc = ctx.getChild(0).getText()
+ if opc == "+":
+ return self.visit(ctx.getChild(1))
+ if opc == "-":
+ return - self.visit(ctx.getChild(1))
+ return 0
+ if ctx.getChildCount() == 1:
+ return self.visit(ctx.getChild(0))
+ return 0
-* deriving your parser from a parser you provide, such as BaseParser
-* implementing utility methods in this BaseParser, such as "isEqualText"
-* adding a "self" field to the Java/C# BaseParser, and initialize it with "this"
+ def visitStart_(self, ctx:ExprParser.Start_Context):
+ for i in range(0, ctx.getChildCount(), 2):
+ print(self.visit(ctx.getChild(i)))
+ return 0
+ ```
-Thanks to the above, you should be able to rewrite the above semantic predicate as follows:
+### Listeners
+
+Antlr listeners perform an LR tree traversal. `enter` and `exit` methods are
+called during the tranversal. A parse tree node is visited twice, first for
+the `enter` method, then the `exit` method after all children have been walked.
+
+To implement a listener, add the `-listener` option to the `antlr4` command.
+Add a class that inherits from the generated listener
+with code that implements the analysis.
+
+The following example implements an expression evaluator using a listener.
+
+* Driver.py:
+ ```python
+ import sys
+ from antlr4 import *
+ from ExprLexer import ExprLexer
+ from ExprParser import ExprParser
+ from ListenerInterp import ListenerInterp
+
+ def main(argv):
+ input_stream = FileStream(argv[1])
+ lexer = ExprLexer(input_stream)
+ stream = CommonTokenStream(lexer)
+ parser = ExprParser(stream)
+ tree = parser.start_()
+ if parser.getNumberOfSyntaxErrors() > 0:
+ print("syntax errors")
+ else:
+ linterp = ListenerInterp()
+ walker = ParseTreeWalker()
+ walker.walk(linterp, tree)
+
+ if __name__ == '__main__':
+ main(sys.argv)
+ ```
+ * ListenerInterp.py:
+ ```python
+ import sys
+ from antlr4 import *
+ from ExprParser import ExprParser
+ from ExprListener import ExprListener
+
+ class ListenerInterp(ExprListener):
+ def __init__(self):
+ self.result = {}
+
+ def exitAtom(self, ctx:ExprParser.AtomContext):
+ self.result[ctx] = int(ctx.getText())
+
+ def exitExpr(self, ctx:ExprParser.ExprContext):
+ if ctx.getChildCount() == 3:
+ if ctx.getChild(0).getText() == "(":
+ self.result[ctx] = self.result[ctx.getChild(1)]
+ else:
+ opc = ctx.getChild(1).getText()
+ v1 = self.result[ctx.getChild(0)]
+ v2 = self.result[ctx.getChild(2)]
+ if opc == "+":
+ self.result[ctx] = v1 + v2
+ elif opc == "-":
+ self.result[ctx] = v1 - v2
+ elif opc == "*":
+ self.result[ctx] = v1 * v2
+ elif opc == "/":
+ self.result[ctx] = v1 / v2
+ else:
+ ctx.result[ctx] = 0
+ elif ctx.getChildCount() == 2:
+ opc = ctx.getChild(0).getText()
+ if opc == "+":
+ v = self.result[ctx.getChild(1)]
+ self.result[ctx] = v
+ elif opc == "-":
+ v = self.result[ctx.getChild(1)]
+ self.result[ctx] = - v
+ elif ctx.getChildCount() == 1:
+ self.result[ctx] = self.result[ctx.getChild(0)]
+
+ def exitStart_(self, ctx:ExprParser.Start_Context):
+ for i in range(0, ctx.getChildCount(), 2):
+ print(self.result[ctx.getChild(i)])
+ ```
+
+Further information can be found from the ANTLR 4 definitive guide.
+
+## Examples
+
+The examples from the ANTLR 4 book converted to Python are [here](https://github.com/jszheng/py3antlr4book).
-```
-ID {$self.isEqualText($text,"test")}?
-```
+There are many examples of grammars that target the Python3 target in the
+[grammars-v4 Github repository](https://github.com/antlr/grammars-v4).
diff --git a/doc/releasing-antlr.md b/doc/releasing-antlr.md
index cd1d0b9b47..6c1ffc516d 100644
--- a/doc/releasing-antlr.md
+++ b/doc/releasing-antlr.md
@@ -2,90 +2,112 @@
## Github
-Create a pre-release or full release at github; [Example 4.5-rc-1](https://github.com/antlr/antlr4/releases/tag/4.5-rc-1).
+### Get dev merged into master
+
+Do this or make a PR:
+
+```bash
+cd ~/antlr/code/antlr4
+git checkout master
+git merge dev
+```
+
+### Turn on DCO Enforcement
+
+As of 4.10.1, we will be using the Linux DCO not the previous contributors license agreement that required signing the file. Now, we use the DCO and contributors must use `-s` on each commit to the branch associated with a pull request.
+
+See [GitHub App DCO](https://github.com/apps/dco).
+
+Make sure this feature is turned on for the `antlr4` repo upon release.
### Delete existing release tag
Wack any existing tag as mvn will create one and it fails if already there.
```
-$ git tag -d 4.8
-$ git push origin :refs/tags/4.8
-$ git push upstream :refs/tags/4.8
+$ git tag -d 4.13.2
+$ git push origin :refs/tags/4.13.2
+$ git push upstream :refs/tags/4.13.2
```
-### Create release candidate tag
+### Go release tags
-```bash
-$ git tag -a 4.8-rc1 -m 'heading towards 4.8'
-$ git push origin 4.8-rc1
-$ git push upstream 4.8-rc1
+*I don't think this is necessary anymore as we have moved it release branch to https://github.com/antlr4-go/antlr*
+
+It seems that [Go needs a `v` in the release git tag](https://go.dev/ref/mod#glos-version) so make sure that we double up with 4.13.2 and v4.13.2.
+
+```
+$ git tag -a runtime/Go/antlr/v4/v4.13.2 -m "Go runtime module only"
+$ git push upstream runtime/Go/antlr/v4/v4.13.2
+$ git push origin runtime/Go/antlr/v4/v4.13.2
```
-## Update submodules
-Make sure you tell git to pull in the submodule (for every clone you do of antlr4):
+## Bump version in code and other files
-```bash
-git submodule init
-```
+There are a number of files that require inversion number be updated.
-Also bump version to 4.8 in `runtime/PHP/src/RuntimeMetaData.php`.
-Update the runtime submodules by running the following command:
+Here is a simple script to display any line from the critical files with, say, `4.11.1` in it. Here's an example run of the script:
```bash
-git submodule update --recursive
-git submodule update --remote --merge # might only need this last one but do both
+~/antlr/code/antlr4 $ python scripts/update_antlr_version.py 4.13.1 4.13.2
+Updating ANTLR version from 4.13.1 to 4.13.2
+Set ANTLR repo root (default ~/antlr/code/antlr4):
+Perform antlr4 `mvn clean` and wipe build dirs Y/N? (default no):
+Ok, not cleaning antlr4 dir
+4.13.1 appears on 2 lines so _not_ updating /tmp/antlr4/runtime/JavaScript/package-lock.json
+4.13.1 not in /tmp/antlr4/doc/releasing-antlr.md
```
-Make sure these changes go back to antlr4 repo:
+Make sure this file doesn't have `-SNAPSHOT` when releasing!
-```bash
-git add runtime/PHP
-git commit -m "Update PHP Runtime to latest version"
-```
-
-## Bump version
-
-Edit the repository looking for 4.5 or whatever and update it. Bump version in the following files:
-
- * runtime/Java/src/org/antlr/v4/runtime/RuntimeMetaData.java
- * runtime/Python2/setup.py
- * runtime/Python2/src/antlr4/Recognizer.py
- * runtime/Python3/setup.py
- * runtime/Python3/src/antlr4/Recognizer.py
- * runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs
- * runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Antlr4.Runtime.dotnet.csproj
- * runtime/JavaScript/package.json
- * runtime/JavaScript/src/antlr4/Recognizer.js
- * runtime/Cpp/VERSION
- * runtime/Cpp/runtime/src/RuntimeMetaData.cpp
- * runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake
- * runtime/Cpp/demo/generate.cmd
- * runtime/Go/antlr/recognizer.go
- * runtime/Swift/Antlr4/org/antlr/v4/runtime/RuntimeMetaData.swift
- * tool/src/org/antlr/v4/codegen/target/GoTarget.java
- * tool/src/org/antlr/v4/codegen/target/CppTarget.java
- * tool/src/org/antlr/v4/codegen/target/CSharpTarget.java
- * tool/src/org/antlr/v4/codegen/target/JavaScriptTarget.java
- * tool/src/org/antlr/v4/codegen/target/Python2Target.java
- * tool/src/org/antlr/v4/codegen/target/Python3Target.java
- * tool/src/org/antlr/v4/codegen/target/SwiftTarget.java
- * tool/src/org/antlr/v4/codegen/Target.java
- * tool/resources/org/antlr/v4/tool/templates/codegen/Swift/Swift.stg
-
-Here is a simple script to display any line from the critical files with, say, `4.5` in it:
+```
+runtime/Java/src/org/antlr/v4/runtime/RuntimeMetaData.java
+```
+
+It's also worth doing a quick check to see if you find any other references to a version:
```bash
-find tool runtime -type f -exec grep -l '4\.6' {} \;
+mvn clean
+find . -type f -exec grep -l '4\.12.0' {} \; | grep -v -E '\.o|\.a|\.jar|\.dylib|node_modules/|\.class|tests/|CHANGELOG|\.zip|\.gz|.iml|.svg'
```
Commit to repository.
-## Building
+### PHP runtime
+
+We only have to copy the PHP runtime into the ANTLR repository to run the unittests. But, we still need to bump the version to 4.13.2 in `~/antlr/code/antlr-php-runtime/src/RuntimeMetaData.php` in the separate repository, commit, and push.
-ugh. apparently you have to `mvn install` and then `mvn compile` or some such or subdir pom.xml's won't see the latest runtime build.
+```
+cd ~/antlr/code/antlr-php-runtime/src
+git checkout dev # Should be the default
+git pull origin dev
+... vi RuntimeMetaData.php ...
+git commit -a -m "Update PHP Runtime to latest version"
+git push origin dev
+git checkout master
+git pull origin master
+git merge dev
+git push origin master
+```
+
+## Build XPath parsers
+
+This section addresses a [circular dependency regarding XPath](https://github.com/antlr/antlr4/issues/3600). In the java target I avoided a circular dependency (gen 4.13.2 parser for XPath using 4.13.2 which needs it to build) by hand building the parser: runtime/Java/src/org/antlr/v4/runtime/tree/xpath/XPath.java. Probably we won't have to rerun this for the patch releases, just major ones that alter the ATN serialization.
+
+```bash
+cd ~/antlr/code/antlr4/runtime/Cpp/runtime/src/tree/xpath
+java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.13.2-SNAPSHOT/antlr4-4.13.2-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Cpp XPathLexer.g4
+
+cd ~/antlr/code/antlr4/runtime/CSharp/src/Tree/Xpath
+java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.13.2-SNAPSHOT/antlr4-4.13.2-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=CSharp XPathLexer.g4
+
+cd ~/antlr/code/antlr4/runtime/Python3/tests/expr
+java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.13.2-SNAPSHOT/antlr4-4.13.2-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Python3 Expr.g4
+cd ~/antlr/code/antlr4/runtime/Python3/src/antlr4/xpath
+java -cp ":/Users/parrt/.m2/repository/org/antlr/antlr4/4.13.2-SNAPSHOT/antlr4-4.13.2-SNAPSHOT-complete.jar:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Python3 XPathLexer.g4
+```
## Maven Repository Settings
@@ -134,23 +156,14 @@ Here is the file template
## Maven deploy snapshot
-The goal is to get a snapshot, such as `4.8-SNAPSHOT`, to the staging server: [antlr4 tool](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4) and [antlr4 java runtime](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-runtime).
+The goal is to get a snapshot, such as `4.13.2-SNAPSHOT`, to the staging server: [antlr4 tool](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4/4.13.2-SNAPSHOT/) and [antlr4 java runtime](https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-runtime/4.13.2-SNAPSHOT/).
Do this:
```bash
+$ mvn install -DskipTests # seems required to get the jar files visible to maven
$ mvn deploy -DskipTests
...
-[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ antlr4-tool-testsuite ---
-Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.8-SNAPSHOT/maven-metadata.xml
-Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.8-SNAPSHOT/antlr4-tool-testsuite-4.8-20161211.173752-1.jar
-Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.8-SNAPSHOT/antlr4-tool-testsuite-4.8-20161211.173752-1.jar (3 KB at 3.4 KB/sec)
-Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.8-SNAPSHOT/antlr4-tool-testsuite-4.8-20161211.173752-1.pom
-Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.8-SNAPSHOT/antlr4-tool-testsuite-4.8-20161211.173752-1.pom (3 KB at 6.5 KB/sec)
-Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml
-Downloaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml (371 B at 1.4 KB/sec)
-Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.8-SNAPSHOT/maven-metadata.xml
-Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/4.8-SNAPSHOT/maven-metadata.xml (774 B at 1.8 KB/sec)
Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml
Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antlr4-tool-testsuite/maven-metadata.xml (388 B at 0.9 KB/sec)
[INFO] ------------------------------------------------------------------------
@@ -162,7 +175,7 @@ Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antl
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 6.547 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 2.519 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 2.385 s]
-[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [ 15.276 s]
+[INFO] ANTLR 4 Runtime Tests (4th generation) ............. SUCCESS [ 15.276 s]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 2.233 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
@@ -178,60 +191,54 @@ Uploaded: https://oss.sonatype.org/content/repositories/snapshots/org/antlr/antl
The maven deploy lifecycle phased deploys the artifacts and the poms for the ANTLR project to the [sonatype remote staging server](https://oss.sonatype.org/content/repositories/snapshots/).
```bash
-export JAVA_HOME=`/usr/libexec/java_home -v 1.7`; mvn deploy -DskipTests
+mvn deploy -DskipTests
```
-With JDK 1.7 (not 6 or 8), do this:
+Make sure `gpg` is installed (`brew install gpg` on mac). Also must [create a key and publish it](https://blog.sonatype.com/2010/01/how-to-generate-pgp-signatures-with-maven/) then update `.m2/settings` to use that public key.
+
+Then:
```bash
-export JAVA_HOME=`/usr/libexec/java_home -v 1.7`; mvn release:prepare -Darguments="-DskipTests"
+mvn release:prepare -Darguments="-DskipTests"
```
-Hm...per https://github.com/keybase/keybase-issues/issues/1712 we need this to make gpg work:
+Hmm...per https://github.com/keybase/keybase-issues/issues/1712 we need this to make gpg work:
```bash
export GPG_TTY=$(tty)
```
-Side note to set jdk 1.7 on os x:
+You should see 0x37 in generated .class files after 0xCAFEBABE; see [Java SE 11 = 55 (0x37 hex)](https://en.wikipedia.org/wiki/Java_class_file):
```bash
-alias java='/Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home/bin/java'
-alias javac='/Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home/bin/javac'
-alias javadoc='/Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home/bin/javadoc'
-alias jar='/Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home/bin/jar'
-export JAVA_HOME=`/usr/libexec/java_home -v 1.7`
-```
-
-But I think just this on front of mvn works:
-
-```
-export JAVA_HOME=`/usr/libexec/java_home -v 1.7`; mvn ...
+~/antlr/code/antlr4 $ od -h tool/target/classes/org/antlr/v4/Tool.class |head -1
+0000000 feca beba 0000 3700 ed04 0207 0a9d 0100
+ ^^
```
-You should see 0x33 in generated .class files after 0xCAFEBABE; see [Java SE 7 = 51 (0x33 hex)](https://en.wikipedia.org/wiki/Java_class_file):
+Also verify run time is 1.8:
```bash
-beast:/tmp/org/antlr/v4 $ od -h Tool.class |head -1
-0000000 feca beba 0000 3300 fa04 0207 0ab8 0100
+od -h runtime/Java/target/classes/org/antlr/v4/runtime/Token.class | head -1
+0000000 feca beba 0000 3400 2500 0007 0722 2300
```
It will start out by asking you the version number:
```
...
-What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.8: : 4.8
-What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.8: :
-What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.8: :
-What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.8: :
-What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.8: :
-What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.8: :
-What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.8: : 4.8
-What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.8.1-SNAPSHOT:
+What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.13.2: : 4.13.2
+What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.13.2: :
+What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.13.2: :
+What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.13.2: :
+What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.13.2: :
+What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.13.2: :
+What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.13.2: : 4.13.2
+What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.13.3-SNAPSHOT:
...
```
-Maven will go through your pom.xml files to update versions from 4.8-SNAPSHOT to 4.8 for release and then to 4.8.1-SNAPSHOT after release, which is done with:
+Maven will go through your pom.xml files to update versions from 4.13.2-SNAPSHOT to 4.13.2 for release and then to 4.13.2-SNAPSHOT after release, which is done with:
```bash
mvn release:perform -Darguments="-DskipTests"
@@ -245,97 +252,55 @@ Now, go here:
and on the left click "Staging Repositories". You click the staging repo and close it, then you refresh, click it and release it. It's done when you see it here:
- [https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.8-1/antlr4-runtime-4.8-1.jar](https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.8-1/antlr4-runtime-4.8-1.jar)
-
-All releases should be here: https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/
-
-Copy the jars to antlr.org site and update download/index.html
-
-```bash
-cp ~/.m2/repository/org/antlr/antlr4-runtime/4.8/antlr4-runtime-4.8.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.8.jar
-cp ~/.m2/repository/org/antlr/antlr4/4.8/antlr4-4.8-complete.jar ~/antlr/sites/website-antlr4/download/antlr-4.8-complete.jar
-cd ~/antlr/sites/website-antlr4/download
-git add antlr-4.8-complete.jar
-git add antlr-runtime-4.8.jar
-```
-
-Update on site:
+ [https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.13.2/antlr4-runtime-4.13.2.jar](https://oss.sonatype.org/service/local/repositories/releases/content/org/antlr/antlr4-runtime/4.13.2/antlr4-runtime-4.13.2.jar)
-* download.html
-* index.html
-* api/index.html
-* download/index.html
-* scripts/topnav.js
-
-```
-git commit -a -m 'add 4.8 jars'
-git push origin gh-pages
-```
+All releases should be here: [https://repo1.maven.org/maven2/org/antlr/antlr4-runtime](https://repo1.maven.org/maven2/org/antlr/antlr4-runtime).
## Deploying Targets
### JavaScript
-```bash
-cd runtime/JavaScript
-# git add, commit, push
-```
-
**Push to npm**
+(I think this has to be run before the unit test can run locally as it installs the global lib)
+
```bash
-cd runtime/JavaScript
-npm login
-npm publish antlr4
+cd ~/antlr/code/antlr4/runtime/JavaScript
+rm -rf node_modules # seems we might need this later but try it here
+npm update
+npm install
+npm run build
+npm login # asks for username/password/2FA (npmjs.com)
+npm publish # don't put antlr4 on there or it will try to push the old version for some reason
```
-Move target to website
+Move (and zip) target to website:
```bash
-npm run build
-cp /dist/antlr4.js ~/antlr/sites/website-antlr4/download
+cd src
+zip -r ~/antlr/sites/website-antlr4/download/antlr-javascript-runtime-4.13.2.zip .
```
### CSharp
-Now we have [appveyor create artifact](https://ci.appveyor.com/project/parrt/antlr4/build/artifacts). Go to [nuget](https://www.nuget.org/packages/manage/upload) to upload the `.nupkg`.
-
-### Publishing to Nuget from Windows
+As of writing, you can only release from a Windows box, because Visual Studio for Mac can only build the netstandard2.0 version
**Install the pre-requisites**
-Of course you need Mono and `nuget` to be installed. On mac:
+You need 'msbuild' and `nuget` to be installed.
-- .NET build tools - can be loaded from [here](https://www.visualstudio.com/downloads/)
-- nuget - download [nuget.exe](https://www.nuget.org/downloads)
-- dotnet - follow [the instructions here](https://www.microsoft.com/net/core)
+**Creating the signed assembly**
-Alternatively, you can install Visual Studio 2017 and make sure to check boxes with .NET Core SDK.
+cd ~/antlr/code/antlr4/runtime/CSharp/src
+dotnet build -c Release Antlr4.csproj
-You also need to enable .NET Framework 3.5 support in Windows "Programs and Features".
+check that the bin/Release folder contains both the netstandard2.0 and the net45 builds
+the binaries are already signed, but it's worth double checking
-If everything is ok, the following command will restore nuget packages, build Antlr for .NET Standard and .NET 3.5 and create nuget package:
+sn -v bin/Release/netstandard2.0/Antlr4.Runtime.Standard.dll
+sn -v bin/Release/net45/Antlr4.Runtime.Standard.dll
-```PS
-msbuild /target:restore /target:rebuild /target:pack /property:Configuration=Release .\Antlr4.dotnet.sln /verbosity:minimal
-```
-
-This should display something like this:
-
-**Creating and packaging the assembly**
-
-```
-Microsoft (R) Build Engine version 15.4.8.50001 for .NET Framework
-Copyright (C) Microsoft Corporation. All rights reserved.
-
- Restoring packages for C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Antlr4.Runtime.dotnet.csproj...
- Generating MSBuild file C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\obj\Antlr4.Runtime.dotnet.csproj.nuget.g.props.
- Generating MSBuild file C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\obj\Antlr4.Runtime.dotnet.csproj.nuget.g.targets.
- Restore completed in 427.62 ms for C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Antlr4.Runtime.dotnet.csproj.
- Antlr4.Runtime.dotnet -> C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\lib\Release\netstandard1.3\Antlr4.Runtime.Standard.dll
- Antlr4.Runtime.dotnet -> C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\lib\Release\net35\Antlr4.Runtime.Standard.dll
- Successfully created package 'C:\Code\antlr4-fork\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\lib\Release\Antlr4.Runtime.Standard.4.8.2.nupkg'.
-```
+both should say the dll is valid
**Publishing to NuGet**
@@ -345,14 +310,15 @@ As a registered NuGet user, you can then manually upload the package here: [http
Alternately, you can publish from the cmd line. You need to get your NuGet key from [https://www.nuget.org/account#](https://www.nuget.org/account#) and then from the cmd line, you can then type:
```cmd
+cd bin/Release
nuget push Antlr4.Runtime.Standard..nupkg -Source https://www.nuget.org/api/v2/package
```
-Nuget packages are also accessible as artifacts of [AppVeyor builds](https://ci.appveyor.com/project/parrt/antlr4/build/artifacts).
-
### Python
-The Python targets get deployed with `setup.py`. First, set up `~/.pypirc` with tight privileges:
+The Python target gets deployed with `twine` for Python 3.
+
+First, set up `~/.pypirc` with tight privileges:
```bash
beast:~ $ ls -l ~/.pypirc
@@ -374,20 +340,13 @@ username: parrt
password: xxx
```
-Then run the usual python set up stuff:
-
-```bash
-cd ~/antlr/code/antlr4/runtime/Python2
-# assume you have ~/.pypirc set up
-python2 setup.py sdist upload
-```
-
-and do again for Python 3 target
+Then run the python build and upload:
```bash
cd ~/antlr/code/antlr4/runtime/Python3
+python -m build
# assume you have ~/.pypirc set up
-python3 setup.py sdist upload
+twine upload dist/antlr4_python3_runtime-4.13.2.tar.gz dist/antlr4_python3_runtime-4.13.2-py3-none-any.whl
```
There are links to the artifacts in [download.html](http://www.antlr.org/download.html) already.
@@ -408,49 +367,112 @@ For each platform there's a deployment script which generates zip archives and c
On a Mac (with XCode 7+ installed):
```bash
-cd runtime/Cpp
+cd ~/antlr/code/antlr4/runtime/Cpp
+rm CMakeCache.txt # otherwise can't find some include files
./deploy-macos.sh
-cp antlr4-cpp-runtime-macos.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.8-macos.zip
+cp antlr4-cpp-runtime-macos.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.13.2-macos.zip
```
On any Mac or Linux machine:
```bash
-cd runtime/Cpp
+cd ~/antlr/code/antlr4/runtime/Cpp
./deploy-source.sh
-cp antlr4-cpp-runtime-source.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.8-source.zip
+cp antlr4-cpp-runtime-source.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.13.2-source.zip
```
On a Windows machine the build scripts checks if VS 2017 and/or VS 2019 are installed and builds binaries for each, if found. This script requires 7z to be installed (http://7-zip.org then do `set PATH=%PATH%;C:\Program Files\7-Zip\` from DOS not powershell).
```bash
-cd runtime/Cpp
+cd ~/antlr/code/antlr4/runtime/Cpp
deploy-windows.cmd Community
-cp antlr4-cpp-runtime-vs2019.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.8-vs2019.zip
+cp antlr4-cpp-runtime-vs2019.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.13.2-vs2019.zip
```
Move target to website (**_rename to a specific ANTLR version first if needed_**):
```bash
pushd ~/antlr/sites/website-antlr4/download
-# vi index.html
-git add antlr4cpp-runtime-4.8-macos.zip
-git add antlr4cpp-runtime-4.8-windows.zip
-git add antlr4cpp-runtime-4.8-source.zip
+git add antlr4-cpp-runtime-4.13.2-macos.zip
+git add antlr4-cpp-runtime-4.13.2-windows.zip
+git add antlr4-cpp-runtime-4.13.2-source.zip
git commit -a -m 'update C++ runtime'
git push origin gh-pages
popd
```
-## Update javadoc for runtime and tool
+### Dart
+
+Install Dart SDK from https://dart.dev/get-dart
-First, gen javadoc:
+Push to pub.dev
```bash
-$ cd antlr4
-$ mvn -DskipTests javadoc:jar install
+cd ~/antlr/code/antlr4/runtime/Dart
+dart pub publish
+```
+
+It will warn that no change log found for the new version.
+Otherwise enter `N` to ignore the warning.
+
+## Update website
+
+### javadoc for runtime and tool
+
+Jars are in:
+
+```
+~/.m2/repository/org/antlr/antlr4-runtime/4.13.2/antlr4-runtime-4.13.2
```
+### Update version and copy jars / api
+
+Copy javadoc and java jars to website using this script:
+
+```bash
+cd ~/antlr/code/antlr4
+python scripts/deploy_to_website.py 4.13.1 4.13.2
+```
+
+Output:
+
+```bash
+Updating ANTLR version from 4.13.1 to 4.13.2
+Set ANTLR website root (default /Users/parrt/antlr/sites/website-antlr4):
+Version string updated. Please commit/push:
+Javadoc copied:
+ api/Java updated from antlr4-runtime-4.13.2-javadoc.jar
+ api/JavaTool updated from antlr4-4.13.2-javadoc.jar
+Jars copied:
+ antlr-4.13.2-complete.jar
+ antlr-runtime-4.13.2.jar
+
+Please look for and add new api files!!
+Then MANUALLY commit/push:
+
+git commit -a -m 'Update website, javadoc, jars to 4.13.2'
+git push origin gh-pages
+```
+
+
+
+Once it's done, you must do the following manually:
+
+```
+cd ~/antlr/sites/website-antlr4
+git commit -a -m 'Update website, javadoc, jars to 4.13.2'
+git push origin gh-pages
+```
+
+
+
+## Get fresh dev branch
+
+```bash
+git checkout master
+git pull upstream master
+git checkout dev
+git pull upstream dev
+git merge master
+git push origin dev
+git push upstream dev
+```
-## Update Intellij plug-in
+## Other updates
-Rebuild antlr plugin with new antlr jar.
+* Rebuild antlr Intellij plug-in with new antlr jar.
+* Cut release notes in github
+* Update lab.antlr.org
diff --git a/doc/resources/CaseChangingCharStream.cs b/doc/resources/CaseChangingCharStream.cs
deleted file mode 100644
index 9f73a03826..0000000000
--- a/doc/resources/CaseChangingCharStream.cs
+++ /dev/null
@@ -1,105 +0,0 @@
-/* Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
- * Use of this file is governed by the BSD 3-clause license that
- * can be found in the LICENSE.txt file in the project root.
- */
-using System;
-using Antlr4.Runtime.Misc;
-
-namespace Antlr4.Runtime
-{
- ///
- /// This class supports case-insensitive lexing by wrapping an existing
- /// and forcing the lexer to see either upper or
- /// lowercase characters. Grammar literals should then be either upper or
- /// lower case such as 'BEGIN' or 'begin'. The text of the character
- /// stream is unaffected. Example: input 'BeGiN' would match lexer rule
- /// 'BEGIN' if constructor parameter upper=true but getText() would return
- /// 'BeGiN'.
- ///
- public class CaseChangingCharStream : ICharStream
- {
- private ICharStream stream;
- private bool upper;
-
- ///
- /// Constructs a new CaseChangingCharStream wrapping the given forcing
- /// all characters to upper case or lower case.
- ///
- /// The stream to wrap.
- /// If true force each symbol to upper case, otherwise force to lower.
- public CaseChangingCharStream(ICharStream stream, bool upper)
- {
- this.stream = stream;
- this.upper = upper;
- }
-
- public int Index
- {
- get
- {
- return stream.Index;
- }
- }
-
- public int Size
- {
- get
- {
- return stream.Size;
- }
- }
-
- public string SourceName
- {
- get
- {
- return stream.SourceName;
- }
- }
-
- public void Consume()
- {
- stream.Consume();
- }
-
- [return: NotNull]
- public string GetText(Interval interval)
- {
- return stream.GetText(interval);
- }
-
- public int LA(int i)
- {
- int c = stream.LA(i);
-
- if (c <= 0)
- {
- return c;
- }
-
- char o = (char)c;
-
- if (upper)
- {
- return (int)char.ToUpperInvariant(o);
- }
-
- return (int)char.ToLowerInvariant(o);
- }
-
- public int Mark()
- {
- return stream.Mark();
- }
-
- public void Release(int marker)
- {
- stream.Release(marker);
- }
-
- public void Seek(int index)
- {
- stream.Seek(index);
- }
- }
-}
diff --git a/doc/resources/CaseChangingCharStream.java b/doc/resources/CaseChangingCharStream.java
deleted file mode 100644
index d069d0188a..0000000000
--- a/doc/resources/CaseChangingCharStream.java
+++ /dev/null
@@ -1,81 +0,0 @@
-package org.antlr.v4.runtime;
-
-import org.antlr.v4.runtime.misc.Interval;
-
-/**
- * This class supports case-insensitive lexing by wrapping an existing
- * {@link CharStream} and forcing the lexer to see either upper or
- * lowercase characters. Grammar literals should then be either upper or
- * lower case such as 'BEGIN' or 'begin'. The text of the character
- * stream is unaffected. Example: input 'BeGiN' would match lexer rule
- * 'BEGIN' if constructor parameter upper=true but getText() would return
- * 'BeGiN'.
- */
-public class CaseChangingCharStream implements CharStream {
-
- final CharStream stream;
- final boolean upper;
-
- /**
- * Constructs a new CaseChangingCharStream wrapping the given {@link CharStream} forcing
- * all characters to upper case or lower case.
- * @param stream The stream to wrap.
- * @param upper If true force each symbol to upper case, otherwise force to lower.
- */
- public CaseChangingCharStream(CharStream stream, boolean upper) {
- this.stream = stream;
- this.upper = upper;
- }
-
- @Override
- public String getText(Interval interval) {
- return stream.getText(interval);
- }
-
- @Override
- public void consume() {
- stream.consume();
- }
-
- @Override
- public int LA(int i) {
- int c = stream.LA(i);
- if (c <= 0) {
- return c;
- }
- if (upper) {
- return Character.toUpperCase(c);
- }
- return Character.toLowerCase(c);
- }
-
- @Override
- public int mark() {
- return stream.mark();
- }
-
- @Override
- public void release(int marker) {
- stream.release(marker);
- }
-
- @Override
- public int index() {
- return stream.index();
- }
-
- @Override
- public void seek(int index) {
- stream.seek(index);
- }
-
- @Override
- public int size() {
- return stream.size();
- }
-
- @Override
- public String getSourceName() {
- return stream.getSourceName();
- }
-}
diff --git a/doc/resources/CaseChangingStream.js b/doc/resources/CaseChangingStream.js
deleted file mode 100644
index 3af1ad6127..0000000000
--- a/doc/resources/CaseChangingStream.js
+++ /dev/null
@@ -1,65 +0,0 @@
-//
-/* Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
- * Use of this file is governed by the BSD 3-clause license that
- * can be found in the LICENSE.txt file in the project root.
- */
-//
-
-function CaseChangingStream(stream, upper) {
- this._stream = stream;
- this._upper = upper;
-}
-
-CaseChangingStream.prototype.LA = function(offset) {
- var c = this._stream.LA(offset);
- if (c <= 0) {
- return c;
- }
- return String.fromCodePoint(c)[this._upper ? "toUpperCase" : "toLowerCase"]().codePointAt(0);
-};
-
-CaseChangingStream.prototype.reset = function() {
- return this._stream.reset();
-};
-
-CaseChangingStream.prototype.consume = function() {
- return this._stream.consume();
-};
-
-CaseChangingStream.prototype.LT = function(offset) {
- return this._stream.LT(offset);
-};
-
-CaseChangingStream.prototype.mark = function() {
- return this._stream.mark();
-};
-
-CaseChangingStream.prototype.release = function(marker) {
- return this._stream.release(marker);
-};
-
-CaseChangingStream.prototype.seek = function(_index) {
- return this._stream.seek(_index);
-};
-
-CaseChangingStream.prototype.getText = function(start, stop) {
- return this._stream.getText(start, stop);
-};
-
-CaseChangingStream.prototype.toString = function() {
- return this._stream.toString();
-};
-
-Object.defineProperty(CaseChangingStream.prototype, "index", {
- get: function() {
- return this._stream.index;
- }
-});
-
-Object.defineProperty(CaseChangingStream.prototype, "size", {
- get: function() {
- return this._stream.size;
- }
-});
-
-exports.CaseChangingStream = CaseChangingStream;
diff --git a/doc/resources/CaseChangingStream.py b/doc/resources/CaseChangingStream.py
deleted file mode 100644
index 6d2815de41..0000000000
--- a/doc/resources/CaseChangingStream.py
+++ /dev/null
@@ -1,13 +0,0 @@
-class CaseChangingStream():
- def __init__(self, stream, upper):
- self._stream = stream
- self._upper = upper
-
- def __getattr__(self, name):
- return self._stream.__getattribute__(name)
-
- def LA(self, offset):
- c = self._stream.LA(offset)
- if c <= 0:
- return c
- return ord(chr(c).upper() if self._upper else chr(c).lower())
diff --git a/doc/resources/case_changing_stream.go b/doc/resources/case_changing_stream.go
deleted file mode 100644
index 5b510fa321..0000000000
--- a/doc/resources/case_changing_stream.go
+++ /dev/null
@@ -1,37 +0,0 @@
-package antlr_resource
-
-import (
- "unicode"
-
- "github.com/antlr/antlr4/runtime/Go/antlr"
-)
-
-// CaseChangingStream wraps an existing CharStream, but upper cases, or
-// lower cases the input before it is tokenized.
-type CaseChangingStream struct {
- antlr.CharStream
-
- upper bool
-}
-
-// NewCaseChangingStream returns a new CaseChangingStream that forces
-// all tokens read from the underlying stream to be either upper case
-// or lower case based on the upper argument.
-func NewCaseChangingStream(in antlr.CharStream, upper bool) *CaseChangingStream {
- return &CaseChangingStream{in, upper}
-}
-
-// LA gets the value of the symbol at offset from the current position
-// from the underlying CharStream and converts it to either upper case
-// or lower case.
-func (is *CaseChangingStream) LA(offset int) int {
- in := is.CharStream.LA(offset)
- if in < 0 {
- // Such as antlr.TokenEOF which is -1
- return in
- }
- if is.upper {
- return int(unicode.ToUpper(rune(in)))
- }
- return int(unicode.ToLower(rune(in)))
-}
diff --git a/doc/resources/worker-base.js b/doc/resources/worker-base.js
deleted file mode 100644
index 3494b39169..0000000000
--- a/doc/resources/worker-base.js
+++ /dev/null
@@ -1,1079 +0,0 @@
-"no use strict";
-(function(e) {
- if (typeof e.window != "undefined" && e.document) return;
- e.console = function() {
- var e = Array.prototype.slice.call(arguments, 0);
- postMessage({
- type: "log",
- data: e
- })
- }, e.console.error = e.console.warn = e.console.log = e.console.trace = e.console, e.window = e, e.ace = e, e.onerror = function(e, t, n, r, i) {
- postMessage({
- type: "error",
- data: {
- message: e,
- file: t,
- line: n,
- col: r,
- stack: i.stack
- }
- })
- }, e.normalizeModule = function(t, n) {
- if (n.indexOf("!") !== -1) {
- var r = n.split("!");
- return e.normalizeModule(t, r[0]) + "!" + e.normalizeModule(t, r[1])
- }
- if (n.charAt(0) == ".") {
- var i = t.split("/")
- .slice(0, -1)
- .join("/");
- n = (i ? i + "/" : "") + n;
- while (n.indexOf(".") !== -1 && s != n) {
- var s = n;
- n = n.replace(/^\.\//, "")
- .replace(/\/\.\//, "/")
- .replace(/[^\/]+\/\.\.\//, "")
- }
- }
- return n
- }, e.require = function(t, n) {
- n || (n = t, t = null);
- if (!n.charAt) throw new Error("worker.js require() accepts only (parentId, id) as arguments");
- n = e.normalizeModule(t, n);
- var r = e.require.modules[n];
- if (r) return r.initialized || (r.initialized = !0, r.exports = r.factory()
- .exports), r.exports;
- var i = n.split("/");
- if (!e.require.tlns) return console.log("unable to load " + n);
- i[0] = e.require.tlns[i[0]] || i[0];
- var s = i.join("/") + ".js";
- return e.require.id = n, importScripts(s), e.require(t, n)
- }, e.require.modules = {}, e.require.tlns = {}, e.define = function(t, n, r) {
- arguments.length == 2 ? (r = n, typeof t != "string" && (n = t, t = e.require.id)) : arguments.length == 1 && (r = t, n = [], t = e.require.id);
- if (typeof r != "function") {
- e.require.modules[t] = {
- exports: r,
- initialized: !0
- };
- return
- }
- n.length || (n = ["require", "exports", "module"]);
- var i = function(n) {
- return e.require(t, n)
- };
- e.require.modules[t] = {
- exports: {},
- factory: function() {
- var e = this,
- t = r.apply(this, n.map(function(t) {
- switch (t) {
- case "require":
- return i;
- case "exports":
- return e.exports;
- case "module":
- return e;
- default:
- return i(t)
- }
- }));
- return t && (e.exports = t), e
- }
- }
- }, e.define.amd = {}, e.initBaseUrls = function(t) {
- require.tlns = t
- }, e.initSender = function() {
- var n = e.require("ace/lib/event_emitter")
- .EventEmitter,
- r = e.require("ace/lib/oop"),
- i = function() {};
- return function() {
- r.implement(this, n), this.callback = function(e, t) {
- postMessage({
- type: "call",
- id: t,
- data: e
- })
- }, this.emit = function(e, t) {
- postMessage({
- type: "event",
- name: e,
- data: t
- })
- }
- }.call(i.prototype), new i
- };
- var t = e.main = null,
- n = e.sender = null;
- e.onmessage = function(r) {
- var i = r.data;
- if (i.command) {
- if (!t[i.command]) throw new Error("Unknown command:" + i.command);
- t[i.command].apply(t, i.args)
- } else if (i.init) {
- initBaseUrls(i.tlns), require("ace/lib/es5-shim"), n = e.sender = initSender();
- var s = require(i.module)[i.classname];
- t = e.main = new s(n)
- } else i.event && n && n._signal(i.event, i.data)
- }
-})(this), ace.define("ace/lib/oop", ["require", "exports", "module"], function(e, t, n) {
- "use strict";
- t.inherits = function(e, t) {
- e.super_ = t, e.prototype = Object.create(t.prototype, {
- constructor: {
- value: e,
- enumerable: !1,
- writable: !0,
- configurable: !0
- }
- })
- }, t.mixin = function(e, t) {
- for (var n in t) e[n] = t[n];
- return e
- }, t.implement = function(e, n) {
- t.mixin(e, n)
- }
-}), ace.define("ace/lib/event_emitter", ["require", "exports", "module"], function(e, t, n) {
- "use strict";
- var r = {},
- i = function() {
- this.propagationStopped = !0
- },
- s = function() {
- this.defaultPrevented = !0
- };
- r._emit = r._dispatchEvent = function(e, t) {
- this._eventRegistry || (this._eventRegistry = {}), this._defaultHandlers || (this._defaultHandlers = {});
- var n = this._eventRegistry[e] || [],
- r = this._defaultHandlers[e];
- if (!n.length && !r) return;
- if (typeof t != "object" || !t) t = {};
- t.type || (t.type = e), t.stopPropagation || (t.stopPropagation = i), t.preventDefault || (t.preventDefault = s), n = n.slice();
- for (var o = 0; o < n.length; o++) {
- n[o](t, this);
- if (t.propagationStopped) break
- }
- if (r && !t.defaultPrevented) return r(t, this)
- }, r._signal = function(e, t) {
- var n = (this._eventRegistry || {})[e];
- if (!n) return;
- n = n.slice();
- for (var r = 0; r < n.length; r++) n[r](t, this)
- }, r.once = function(e, t) {
- var n = this;
- t && this.addEventListener(e, function r() {
- n.removeEventListener(e, r), t.apply(null, arguments)
- })
- }, r.setDefaultHandler = function(e, t) {
- var n = this._defaultHandlers;
- n || (n = this._defaultHandlers = {
- _disabled_: {}
- });
- if (n[e]) {
- var r = n[e],
- i = n._disabled_[e];
- i || (n._disabled_[e] = i = []), i.push(r);
- var s = i.indexOf(t);
- s != -1 && i.splice(s, 1)
- }
- n[e] = t
- }, r.removeDefaultHandler = function(e, t) {
- var n = this._defaultHandlers;
- if (!n) return;
- var r = n._disabled_[e];
- if (n[e] == t) {
- var i = n[e];
- r && this.setDefaultHandler(e, r.pop())
- } else if (r) {
- var s = r.indexOf(t);
- s != -1 && r.splice(s, 1)
- }
- }, r.on = r.addEventListener = function(e, t, n) {
- this._eventRegistry = this._eventRegistry || {};
- var r = this._eventRegistry[e];
- return r || (r = this._eventRegistry[e] = []), r.indexOf(t) == -1 && r[n ? "unshift" : "push"](t), t
- }, r.off = r.removeListener = r.removeEventListener = function(e, t) {
- this._eventRegistry = this._eventRegistry || {};
- var n = this._eventRegistry[e];
- if (!n) return;
- var r = n.indexOf(t);
- r !== -1 && n.splice(r, 1)
- }, r.removeAllListeners = function(e) {
- this._eventRegistry && (this._eventRegistry[e] = [])
- }, t.EventEmitter = r
-}), ace.define("ace/range", ["require", "exports", "module"], function(e, t, n) {
- "use strict";
- var r = function(e, t) {
- return e.row - t.row || e.column - t.column
- },
- i = function(e, t, n, r) {
- this.start = {
- row: e,
- column: t
- }, this.end = {
- row: n,
- column: r
- }
- };
- (function() {
- this.isEqual = function(e) {
- return this.start.row === e.start.row && this.end.row === e.end.row && this.start.column === e.start.column && this.end.column === e.end.column
- }, this.toString = function() {
- return "Range: [" + this.start.row + "/" + this.start.column + "] -> [" + this.end.row + "/" + this.end.column + "]"
- }, this.contains = function(e, t) {
- return this.compare(e, t) == 0
- }, this.compareRange = function(e) {
- var t, n = e.end,
- r = e.start;
- return t = this.compare(n.row, n.column), t == 1 ? (t = this.compare(r.row, r.column), t == 1 ? 2 : t == 0 ? 1 : 0) : t == -1 ? -2 : (t = this.compare(r.row, r.column), t == -1 ? -1 : t == 1 ? 42 : 0)
- }, this.comparePoint = function(e) {
- return this.compare(e.row, e.column)
- }, this.containsRange = function(e) {
- return this.comparePoint(e.start) == 0 && this.comparePoint(e.end) == 0
- }, this.intersects = function(e) {
- var t = this.compareRange(e);
- return t == -1 || t == 0 || t == 1
- }, this.isEnd = function(e, t) {
- return this.end.row == e && this.end.column == t
- }, this.isStart = function(e, t) {
- return this.start.row == e && this.start.column == t
- }, this.setStart = function(e, t) {
- typeof e == "object" ? (this.start.column = e.column, this.start.row = e.row) : (this.start.row = e, this.start.column = t)
- }, this.setEnd = function(e, t) {
- typeof e == "object" ? (this.end.column = e.column, this.end.row = e.row) : (this.end.row = e, this.end.column = t)
- }, this.inside = function(e, t) {
- return this.compare(e, t) == 0 ? this.isEnd(e, t) || this.isStart(e, t) ? !1 : !0 : !1
- }, this.insideStart = function(e, t) {
- return this.compare(e, t) == 0 ? this.isEnd(e, t) ? !1 : !0 : !1
- }, this.insideEnd = function(e, t) {
- return this.compare(e, t) == 0 ? this.isStart(e, t) ? !1 : !0 : !1
- }, this.compare = function(e, t) {
- return !this.isMultiLine() && e === this.start.row ? t < this.start.column ? -1 : t > this.end.column ? 1 : 0 : e < this.start.row ? -1 : e > this.end.row ? 1 : this.start.row === e ? t >= this.start.column ? 0 : -1 : this.end.row === e ? t <= this.end.column ? 0 : 1 : 0
- }, this.compareStart = function(e, t) {
- return this.start.row == e && this.start.column == t ? -1 : this.compare(e, t)
- }, this.compareEnd = function(e, t) {
- return this.end.row == e && this.end.column == t ? 1 : this.compare(e, t)
- }, this.compareInside = function(e, t) {
- return this.end.row == e && this.end.column == t ? 1 : this.start.row == e && this.start.column == t ? -1 : this.compare(e, t)
- }, this.clipRows = function(e, t) {
- if (this.end.row > t) var n = {
- row: t + 1,
- column: 0
- };
- else if (this.end.row < e) var n = {
- row: e,
- column: 0
- };
- if (this.start.row > t) var r = {
- row: t + 1,
- column: 0
- };
- else if (this.start.row < e) var r = {
- row: e,
- column: 0
- };
- return i.fromPoints(r || this.start, n || this.end)
- }, this.extend = function(e, t) {
- var n = this.compare(e, t);
- if (n == 0) return this;
- if (n == -1) var r = {
- row: e,
- column: t
- };
- else var s = {
- row: e,
- column: t
- };
- return i.fromPoints(r || this.start, s || this.end)
- }, this.isEmpty = function() {
- return this.start.row === this.end.row && this.start.column === this.end.column
- }, this.isMultiLine = function() {
- return this.start.row !== this.end.row
- }, this.clone = function() {
- return i.fromPoints(this.start, this.end)
- }, this.collapseRows = function() {
- return this.end.column == 0 ? new i(this.start.row, 0, Math.max(this.start.row, this.end.row - 1), 0) : new i(this.start.row, 0, this.end.row, 0)
- }, this.toScreenRange = function(e) {
- var t = e.documentToScreenPosition(this.start),
- n = e.documentToScreenPosition(this.end);
- return new i(t.row, t.column, n.row, n.column)
- }, this.moveBy = function(e, t) {
- this.start.row += e, this.start.column += t, this.end.row += e, this.end.column += t
- }
- })
- .call(i.prototype), i.fromPoints = function(e, t) {
- return new i(e.row, e.column, t.row, t.column)
- }, i.comparePoints = r, i.comparePoints = function(e, t) {
- return e.row - t.row || e.column - t.column
- }, t.Range = i
-}), ace.define("ace/anchor", ["require", "exports", "module", "ace/lib/oop", "ace/lib/event_emitter"], function(e, t, n) {
- "use strict";
- var r = e("./lib/oop"),
- i = e("./lib/event_emitter")
- .EventEmitter,
- s = t.Anchor = function(e, t, n) {
- this.$onChange = this.onChange.bind(this), this.attach(e), typeof n == "undefined" ? this.setPosition(t.row, t.column) : this.setPosition(t, n)
- };
- (function() {
- r.implement(this, i), this.getPosition = function() {
- return this.$clipPositionToDocument(this.row, this.column)
- }, this.getDocument = function() {
- return this.document
- }, this.$insertRight = !1, this.onChange = function(e) {
- var t = e.data,
- n = t.range;
- if (n.start.row == n.end.row && n.start.row != this.row) return;
- if (n.start.row > this.row) return;
- if (n.start.row == this.row && n.start.column > this.column) return;
- var r = this.row,
- i = this.column,
- s = n.start,
- o = n.end;
- if (t.action === "insertText")
- if (s.row === r && s.column <= i) {
- if (s.column !== i || !this.$insertRight) s.row === o.row ? i += o.column - s.column : (i -= s.column, r += o.row - s.row)
- } else s.row !== o.row && s.row < r && (r += o.row - s.row);
- else t.action === "insertLines" ? (s.row !== r || i !== 0 || !this.$insertRight) && s.row <= r && (r += o.row - s.row) : t.action === "removeText" ? s.row === r && s.column < i ? o.column >= i ? i = s.column : i = Math.max(0, i - (o.column - s.column)) : s.row !== o.row && s.row < r ? (o.row === r && (i = Math.max(0, i - o.column) + s.column), r -= o.row - s.row) : o.row === r && (r -= o.row - s.row, i = Math.max(0, i - o.column) + s.column) : t.action == "removeLines" && s.row <= r && (o.row <= r ? r -= o.row - s.row : (r = s.row, i = 0));
- this.setPosition(r, i, !0)
- }, this.setPosition = function(e, t, n) {
- var r;
- n ? r = {
- row: e,
- column: t
- } : r = this.$clipPositionToDocument(e, t);
- if (this.row == r.row && this.column == r.column) return;
- var i = {
- row: this.row,
- column: this.column
- };
- this.row = r.row, this.column = r.column, this._signal("change", {
- old: i,
- value: r
- })
- }, this.detach = function() {
- this.document.removeEventListener("change", this.$onChange)
- }, this.attach = function(e) {
- this.document = e || this.document, this.document.on("change", this.$onChange)
- }, this.$clipPositionToDocument = function(e, t) {
- var n = {};
- return e >= this.document.getLength() ? (n.row = Math.max(0, this.document.getLength() - 1), n.column = this.document.getLine(n.row)
- .length) : e < 0 ? (n.row = 0, n.column = 0) : (n.row = e, n.column = Math.min(this.document.getLine(n.row)
- .length, Math.max(0, t))), t < 0 && (n.column = 0), n
- }
- })
- .call(s.prototype)
-}), ace.define("ace/document", ["require", "exports", "module", "ace/lib/oop", "ace/lib/event_emitter", "ace/range", "ace/anchor"], function(e, t, n) {
- "use strict";
- var r = e("./lib/oop"),
- i = e("./lib/event_emitter")
- .EventEmitter,
- s = e("./range")
- .Range,
- o = e("./anchor")
- .Anchor,
- u = function(e) {
- this.$lines = [], e.length === 0 ? this.$lines = [""] : Array.isArray(e) ? this._insertLines(0, e) : this.insert({
- row: 0,
- column: 0
- }, e)
- };
- (function() {
- r.implement(this, i), this.setValue = function(e) {
- var t = this.getLength();
- this.remove(new s(0, 0, t, this.getLine(t - 1)
- .length)), this.insert({
- row: 0,
- column: 0
- }, e)
- }, this.getValue = function() {
- return this.getAllLines()
- .join(this.getNewLineCharacter())
- }, this.createAnchor = function(e, t) {
- return new o(this, e, t)
- }, "aaa".split(/a/)
- .length === 0 ? this.$split = function(e) {
- return e.replace(/\r\n|\r/g, "\n")
- .split("\n")
- } : this.$split = function(e) {
- return e.split(/\r\n|\r|\n/)
- }, this.$detectNewLine = function(e) {
- var t = e.match(/^.*?(\r\n|\r|\n)/m);
- this.$autoNewLine = t ? t[1] : "\n", this._signal("changeNewLineMode")
- }, this.getNewLineCharacter = function() {
- switch (this.$newLineMode) {
- case "windows":
- return "\r\n";
- case "unix":
- return "\n";
- default:
- return this.$autoNewLine || "\n"
- }
- }, this.$autoNewLine = "", this.$newLineMode = "auto", this.setNewLineMode = function(e) {
- if (this.$newLineMode === e) return;
- this.$newLineMode = e, this._signal("changeNewLineMode")
- }, this.getNewLineMode = function() {
- return this.$newLineMode
- }, this.isNewLine = function(e) {
- return e == "\r\n" || e == "\r" || e == "\n"
- }, this.getLine = function(e) {
- return this.$lines[e] || ""
- }, this.getLines = function(e, t) {
- return this.$lines.slice(e, t + 1)
- }, this.getAllLines = function() {
- return this.getLines(0, this.getLength())
- }, this.getLength = function() {
- return this.$lines.length
- }, this.getTextRange = function(e) {
- if (e.start.row == e.end.row) return this.getLine(e.start.row)
- .substring(e.start.column, e.end.column);
- var t = this.getLines(e.start.row, e.end.row);
- t[0] = (t[0] || "")
- .substring(e.start.column);
- var n = t.length - 1;
- return e.end.row - e.start.row == n && (t[n] = t[n].substring(0, e.end.column)), t.join(this.getNewLineCharacter())
- }, this.$clipPosition = function(e) {
- var t = this.getLength();
- return e.row >= t ? (e.row = Math.max(0, t - 1), e.column = this.getLine(t - 1)
- .length) : e.row < 0 && (e.row = 0), e
- }, this.insert = function(e, t) {
- if (!t || t.length === 0) return e;
- e = this.$clipPosition(e), this.getLength() <= 1 && this.$detectNewLine(t);
- var n = this.$split(t),
- r = n.splice(0, 1)[0],
- i = n.length == 0 ? null : n.splice(n.length - 1, 1)[0];
- return e = this.insertInLine(e, r), i !== null && (e = this.insertNewLine(e), e = this._insertLines(e.row, n), e = this.insertInLine(e, i || "")), e
- }, this.insertLines = function(e, t) {
- return e >= this.getLength() ? this.insert({
- row: e,
- column: 0
- }, "\n" + t.join("\n")) : this._insertLines(Math.max(e, 0), t)
- }, this._insertLines = function(e, t) {
- if (t.length == 0) return {
- row: e,
- column: 0
- };
- while (t.length > 2e4) {
- var n = this._insertLines(e, t.slice(0, 2e4));
- t = t.slice(2e4), e = n.row
- }
- var r = [e, 0];
- r.push.apply(r, t), this.$lines.splice.apply(this.$lines, r);
- var i = new s(e, 0, e + t.length, 0),
- o = {
- action: "insertLines",
- range: i,
- lines: t
- };
- return this._signal("change", {
- data: o
- }), i.end
- }, this.insertNewLine = function(e) {
- e = this.$clipPosition(e);
- var t = this.$lines[e.row] || "";
- this.$lines[e.row] = t.substring(0, e.column), this.$lines.splice(e.row + 1, 0, t.substring(e.column, t.length));
- var n = {
- row: e.row + 1,
- column: 0
- },
- r = {
- action: "insertText",
- range: s.fromPoints(e, n),
- text: this.getNewLineCharacter()
- };
- return this._signal("change", {
- data: r
- }), n
- }, this.insertInLine = function(e, t) {
- if (t.length == 0) return e;
- var n = this.$lines[e.row] || "";
- this.$lines[e.row] = n.substring(0, e.column) + t + n.substring(e.column);
- var r = {
- row: e.row,
- column: e.column + t.length
- },
- i = {
- action: "insertText",
- range: s.fromPoints(e, r),
- text: t
- };
- return this._signal("change", {
- data: i
- }), r
- }, this.remove = function(e) {
- e instanceof s || (e = s.fromPoints(e.start, e.end)), e.start = this.$clipPosition(e.start), e.end = this.$clipPosition(e.end);
- if (e.isEmpty()) return e.start;
- var t = e.start.row,
- n = e.end.row;
- if (e.isMultiLine()) {
- var r = e.start.column == 0 ? t : t + 1,
- i = n - 1;
- e.end.column > 0 && this.removeInLine(n, 0, e.end.column), i >= r && this._removeLines(r, i), r != t && (this.removeInLine(t, e.start.column, this.getLine(t)
- .length), this.removeNewLine(e.start.row))
- } else this.removeInLine(t, e.start.column, e.end.column);
- return e.start
- }, this.removeInLine = function(e, t, n) {
- if (t == n) return;
- var r = new s(e, t, e, n),
- i = this.getLine(e),
- o = i.substring(t, n),
- u = i.substring(0, t) + i.substring(n, i.length);
- this.$lines.splice(e, 1, u);
- var a = {
- action: "removeText",
- range: r,
- text: o
- };
- return this._signal("change", {
- data: a
- }), r.start
- }, this.removeLines = function(e, t) {
- return e < 0 || t >= this.getLength() ? this.remove(new s(e, 0, t + 1, 0)) : this._removeLines(e, t)
- }, this._removeLines = function(e, t) {
- var n = new s(e, 0, t + 1, 0),
- r = this.$lines.splice(e, t - e + 1),
- i = {
- action: "removeLines",
- range: n,
- nl: this.getNewLineCharacter(),
- lines: r
- };
- return this._signal("change", {
- data: i
- }), r
- }, this.removeNewLine = function(e) {
- var t = this.getLine(e),
- n = this.getLine(e + 1),
- r = new s(e, t.length, e + 1, 0),
- i = t + n;
- this.$lines.splice(e, 2, i);
- var o = {
- action: "removeText",
- range: r,
- text: this.getNewLineCharacter()
- };
- this._signal("change", {
- data: o
- })
- }, this.replace = function(e, t) {
- e instanceof s || (e = s.fromPoints(e.start, e.end));
- if (t.length == 0 && e.isEmpty()) return e.start;
- if (t == this.getTextRange(e)) return e.end;
- this.remove(e);
- if (t) var n = this.insert(e.start, t);
- else n = e.start;
- return n
- }, this.applyDeltas = function(e) {
- for (var t = 0; t < e.length; t++) {
- var n = e[t],
- r = s.fromPoints(n.range.start, n.range.end);
- n.action == "insertLines" ? this.insertLines(r.start.row, n.lines) : n.action == "insertText" ? this.insert(r.start, n.text) : n.action == "removeLines" ? this._removeLines(r.start.row, r.end.row - 1) : n.action == "removeText" && this.remove(r)
- }
- }, this.revertDeltas = function(e) {
- for (var t = e.length - 1; t >= 0; t--) {
- var n = e[t],
- r = s.fromPoints(n.range.start, n.range.end);
- n.action == "insertLines" ? this._removeLines(r.start.row, r.end.row - 1) : n.action == "insertText" ? this.remove(r) : n.action == "removeLines" ? this._insertLines(r.start.row, n.lines) : n.action == "removeText" && this.insert(r.start, n.text)
- }
- }, this.indexToPosition = function(e, t) {
- var n = this.$lines || this.getAllLines(),
- r = this.getNewLineCharacter()
- .length;
- for (var i = t || 0, s = n.length; i < s; i++) {
- e -= n[i].length + r;
- if (e < 0) return {
- row: i,
- column: e + n[i].length + r
- }
- }
- return {
- row: s - 1,
- column: n[s - 1].length
- }
- }, this.positionToIndex = function(e, t) {
- var n = this.$lines || this.getAllLines(),
- r = this.getNewLineCharacter()
- .length,
- i = 0,
- s = Math.min(e.row, n.length);
- for (var o = t || 0; o < s; ++o) i += n[o].length + r;
- return i + e.column
- }
- })
- .call(u.prototype), t.Document = u
-}), ace.define("ace/lib/lang", ["require", "exports", "module"], function(e, t, n) {
- "use strict";
- t.last = function(e) {
- return e[e.length - 1]
- }, t.stringReverse = function(e) {
- return e.split("")
- .reverse()
- .join("")
- }, t.stringRepeat = function(e, t) {
- var n = "";
- while (t > 0) {
- t & 1 && (n += e);
- if (t >>= 1) e += e
- }
- return n
- };
- var r = /^\s\s*/,
- i = /\s\s*$/;
- t.stringTrimLeft = function(e) {
- return e.replace(r, "")
- }, t.stringTrimRight = function(e) {
- return e.replace(i, "")
- }, t.copyObject = function(e) {
- var t = {};
- for (var n in e) t[n] = e[n];
- return t
- }, t.copyArray = function(e) {
- var t = [];
- for (var n = 0, r = e.length; n < r; n++) e[n] && typeof e[n] == "object" ? t[n] = this.copyObject(e[n]) : t[n] = e[n];
- return t
- }, t.deepCopy = function s(e) {
- if (typeof e != "object" || !e) return e;
- var t;
- if (Array.isArray(e)) {
- t = [];
- for (var n = 0; n < e.length; n++) t[n] = s(e[n]);
- return t
- }
- var r = e.constructor;
- if (r === RegExp) return e;
- t = r();
- for (var n in e) t[n] = s(e[n]);
- return t
- }, t.arrayToMap = function(e) {
- var t = {};
- for (var n = 0; n < e.length; n++) t[e[n]] = 1;
- return t
- }, t.createMap = function(e) {
- var t = Object.create(null);
- for (var n in e) t[n] = e[n];
- return t
- }, t.arrayRemove = function(e, t) {
- for (var n = 0; n <= e.length; n++) t === e[n] && e.splice(n, 1)
- }, t.escapeRegExp = function(e) {
- return e.replace(/([.*+?^${}()|[\]\/\\])/g, "\\$1")
- }, t.escapeHTML = function(e) {
- return e.replace(/&/g, "&")
- .replace(/"/g, """)
- .replace(/'/g, "'")
- .replace(/ 0 || -1) * Math.floor(Math.abs(e))), e
- }
-
- function B(e) {
- var t = typeof e;
- return e === null || t === "undefined" || t === "boolean" || t === "number" || t === "string"
- }
-
- function j(e) {
- var t, n, r;
- if (B(e)) return e;
- n = e.valueOf;
- if (typeof n == "function") {
- t = n.call(e);
- if (B(t)) return t
- }
- r = e.toString;
- if (typeof r == "function") {
- t = r.call(e);
- if (B(t)) return t
- }
- throw new TypeError
- }
- Function.prototype.bind || (Function.prototype.bind = function(t) {
- var n = this;
- if (typeof n != "function") throw new TypeError("Function.prototype.bind called on incompatible " + n);
- var i = u.call(arguments, 1),
- s = function() {
- if (this instanceof s) {
- var e = n.apply(this, i.concat(u.call(arguments)));
- return Object(e) === e ? e : this
- }
- return n.apply(t, i.concat(u.call(arguments)))
- };
- return n.prototype && (r.prototype = n.prototype, s.prototype = new r, r.prototype = null), s
- });
- var i = Function.prototype.call,
- s = Array.prototype,
- o = Object.prototype,
- u = s.slice,
- a = i.bind(o.toString),
- f = i.bind(o.hasOwnProperty),
- l, c, h, p, d;
- if (d = f(o, "__defineGetter__")) l = i.bind(o.__defineGetter__), c = i.bind(o.__defineSetter__), h = i.bind(o.__lookupGetter__), p = i.bind(o.__lookupSetter__);
- if ([1, 2].splice(0)
- .length != 2)
- if (! function() {
- function e(e) {
- var t = new Array(e + 2);
- return t[0] = t[1] = 0, t
- }
- var t = [],
- n;
- t.splice.apply(t, e(20)), t.splice.apply(t, e(26)), n = t.length, t.splice(5, 0, "XXX"), n + 1 == t.length;
- if (n + 1 == t.length) return !0
- }()) Array.prototype.splice = function(e, t) {
- var n = this.length;
- e > 0 ? e > n && (e = n) : e == void 0 ? e = 0 : e < 0 && (e = Math.max(n + e, 0)), e + t < n || (t = n - e);
- var r = this.slice(e, e + t),
- i = u.call(arguments, 2),
- s = i.length;
- if (e === n) s && this.push.apply(this, i);
- else {
- var o = Math.min(t, n - e),
- a = e + o,
- f = a + s - o,
- l = n - a,
- c = n - o;
- if (f < a)
- for (var h = 0; h < l; ++h) this[f + h] = this[a + h];
- else if (f > a)
- for (h = l; h--;) this[f + h] = this[a + h];
- if (s && e === c) this.length = c, this.push.apply(this, i);
- else {
- this.length = c + s;
- for (h = 0; h < s; ++h) this[e + h] = i[h]
- }
- }
- return r
- };
- else {
- var v = Array.prototype.splice;
- Array.prototype.splice = function(e, t) {
- return arguments.length ? v.apply(this, [e === void 0 ? 0 : e, t === void 0 ? this.length - e : t].concat(u.call(arguments, 2))) : []
- }
- }
- Array.isArray || (Array.isArray = function(t) {
- return a(t) == "[object Array]"
- });
- var m = Object("a"),
- g = m[0] != "a" || !(0 in m);
- Array.prototype.forEach || (Array.prototype.forEach = function(t) {
- var n = F(this),
- r = g && a(this) == "[object String]" ? this.split("") : n,
- i = arguments[1],
- s = -1,
- o = r.length >>> 0;
- if (a(t) != "[object Function]") throw new TypeError;
- while (++s < o) s in r && t.call(i, r[s], s, n)
- }), Array.prototype.map || (Array.prototype.map = function(t) {
- var n = F(this),
- r = g && a(this) == "[object String]" ? this.split("") : n,
- i = r.length >>> 0,
- s = Array(i),
- o = arguments[1];
- if (a(t) != "[object Function]") throw new TypeError(t + " is not a function");
- for (var u = 0; u < i; u++) u in r && (s[u] = t.call(o, r[u], u, n));
- return s
- }), Array.prototype.filter || (Array.prototype.filter = function(t) {
- var n = F(this),
- r = g && a(this) == "[object String]" ? this.split("") : n,
- i = r.length >>> 0,
- s = [],
- o, u = arguments[1];
- if (a(t) != "[object Function]") throw new TypeError(t + " is not a function");
- for (var f = 0; f < i; f++) f in r && (o = r[f], t.call(u, o, f, n) && s.push(o));
- return s
- }), Array.prototype.every || (Array.prototype.every = function(t) {
- var n = F(this),
- r = g && a(this) == "[object String]" ? this.split("") : n,
- i = r.length >>> 0,
- s = arguments[1];
- if (a(t) != "[object Function]") throw new TypeError(t + " is not a function");
- for (var o = 0; o < i; o++)
- if (o in r && !t.call(s, r[o], o, n)) return !1;
- return !0
- }), Array.prototype.some || (Array.prototype.some = function(t) {
- var n = F(this),
- r = g && a(this) == "[object String]" ? this.split("") : n,
- i = r.length >>> 0,
- s = arguments[1];
- if (a(t) != "[object Function]") throw new TypeError(t + " is not a function");
- for (var o = 0; o < i; o++)
- if (o in r && t.call(s, r[o], o, n)) return !0;
- return !1
- }), Array.prototype.reduce || (Array.prototype.reduce = function(t) {
- var n = F(this),
- r = g && a(this) == "[object String]" ? this.split("") : n,
- i = r.length >>> 0;
- if (a(t) != "[object Function]") throw new TypeError(t + " is not a function");
- if (!i && arguments.length == 1) throw new TypeError("reduce of empty array with no initial value");
- var s = 0,
- o;
- if (arguments.length >= 2) o = arguments[1];
- else
- do {
- if (s in r) {
- o = r[s++];
- break
- }
- if (++s >= i) throw new TypeError("reduce of empty array with no initial value")
- } while (!0);
- for (; s < i; s++) s in r && (o = t.call(void 0, o, r[s], s, n));
- return o
- }), Array.prototype.reduceRight || (Array.prototype.reduceRight = function(t) {
- var n = F(this),
- r = g && a(this) == "[object String]" ? this.split("") : n,
- i = r.length >>> 0;
- if (a(t) != "[object Function]") throw new TypeError(t + " is not a function");
- if (!i && arguments.length == 1) throw new TypeError("reduceRight of empty array with no initial value");
- var s, o = i - 1;
- if (arguments.length >= 2) s = arguments[1];
- else
- do {
- if (o in r) {
- s = r[o--];
- break
- }
- if (--o < 0) throw new TypeError("reduceRight of empty array with no initial value")
- } while (!0);
- do o in this && (s = t.call(void 0, s, r[o], o, n)); while (o--);
- return s
- });
- if (!Array.prototype.indexOf || [0, 1].indexOf(1, 2) != -1) Array.prototype.indexOf = function(t) {
- var n = g && a(this) == "[object String]" ? this.split("") : F(this),
- r = n.length >>> 0;
- if (!r) return -1;
- var i = 0;
- arguments.length > 1 && (i = H(arguments[1])), i = i >= 0 ? i : Math.max(0, r + i);
- for (; i < r; i++)
- if (i in n && n[i] === t) return i;
- return -1
- };
- if (!Array.prototype.lastIndexOf || [0, 1].lastIndexOf(0, -3) != -1) Array.prototype.lastIndexOf = function(t) {
- var n = g && a(this) == "[object String]" ? this.split("") : F(this),
- r = n.length >>> 0;
- if (!r) return -1;
- var i = r - 1;
- arguments.length > 1 && (i = Math.min(i, H(arguments[1]))), i = i >= 0 ? i : r - Math.abs(i);
- for (; i >= 0; i--)
- if (i in n && t === n[i]) return i;
- return -1
- };
- Object.getPrototypeOf || (Object.getPrototypeOf = function(t) {
- return t.__proto__ || (t.constructor ? t.constructor.prototype : o)
- });
- if (!Object.getOwnPropertyDescriptor) {
- var y = "Object.getOwnPropertyDescriptor called on a non-object: ";
- Object.getOwnPropertyDescriptor = function(t, n) {
- if (typeof t != "object" && typeof t != "function" || t === null) throw new TypeError(y + t);
- if (!f(t, n)) return;
- var r, i, s;
- r = {
- enumerable: !0,
- configurable: !0
- };
- if (d) {
- var u = t.__proto__;
- t.__proto__ = o;
- var i = h(t, n),
- s = p(t, n);
- t.__proto__ = u;
- if (i || s) return i && (r.get = i), s && (r.set = s), r
- }
- return r.value = t[n], r
- }
- }
- Object.getOwnPropertyNames || (Object.getOwnPropertyNames = function(t) {
- return Object.keys(t)
- });
- if (!Object.create) {
- var b;
- Object.prototype.__proto__ === null ? b = function() {
- return {
- __proto__: null
- }
- } : b = function() {
- var e = {};
- for (var t in e) e[t] = null;
- return e.constructor = e.hasOwnProperty = e.propertyIsEnumerable = e.isPrototypeOf = e.toLocaleString = e.toString = e.valueOf = e.__proto__ = null, e
- }, Object.create = function(t, n) {
- var r;
- if (t === null) r = b();
- else {
- if (typeof t != "object") throw new TypeError("typeof prototype[" + typeof t + "] != 'object'");
- var i = function() {};
- i.prototype = t, r = new i, r.__proto__ = t
- }
- return n !== void 0 && Object.defineProperties(r, n), r
- }
- }
- if (Object.defineProperty) {
- var E = w({}),
- S = typeof document == "undefined" || w(document.createElement("div"));
- if (!E || !S) var x = Object.defineProperty
- }
- if (!Object.defineProperty || x) {
- var T = "Property description must be an object: ",
- N = "Object.defineProperty called on non-object: ",
- C = "getters & setters can not be defined on this javascript engine";
- Object.defineProperty = function(t, n, r) {
- if (typeof t != "object" && typeof t != "function" || t === null) throw new TypeError(N + t);
- if (typeof r != "object" && typeof r != "function" || r === null) throw new TypeError(T + r);
- if (x) try {
- return x.call(Object, t, n, r)
- } catch (i) {}
- if (f(r, "value"))
- if (d && (h(t, n) || p(t, n))) {
- var s = t.__proto__;
- t.__proto__ = o, delete t[n], t[n] = r.value, t.__proto__ = s
- } else t[n] = r.value;
- else {
- if (!d) throw new TypeError(C);
- f(r, "get") && l(t, n, r.get), f(r, "set") && c(t, n, r.set)
- }
- return t
- }
- }
- Object.defineProperties || (Object.defineProperties = function(t, n) {
- for (var r in n) f(n, r) && Object.defineProperty(t, r, n[r]);
- return t
- }), Object.seal || (Object.seal = function(t) {
- return t
- }), Object.freeze || (Object.freeze = function(t) {
- return t
- });
- try {
- Object.freeze(function() {})
- } catch (k) {
- Object.freeze = function(t) {
- return function(n) {
- return typeof n == "function" ? n : t(n)
- }
- }(Object.freeze)
- }
- Object.preventExtensions || (Object.preventExtensions = function(t) {
- return t
- }), Object.isSealed || (Object.isSealed = function(t) {
- return !1
- }), Object.isFrozen || (Object.isFrozen = function(t) {
- return !1
- }), Object.isExtensible || (Object.isExtensible = function(t) {
- if (Object(t) === t) throw new TypeError;
- var n = "";
- while (f(t, n)) n += "?";
- t[n] = !0;
- var r = f(t, n);
- return delete t[n], r
- });
- if (!Object.keys) {
- var L = !0,
- A = ["toString", "toLocaleString", "valueOf", "hasOwnProperty", "isPrototypeOf", "propertyIsEnumerable", "constructor"],
- O = A.length;
- for (var M in {
- toString: null
- }) L = !1;
- Object.keys = function I(e) {
- if (typeof e != "object" && typeof e != "function" || e === null) throw new TypeError("Object.keys called on a non-object");
- var I = [];
- for (var t in e) f(e, t) && I.push(t);
- if (L)
- for (var n = 0, r = O; n < r; n++) {
- var i = A[n];
- f(e, i) && I.push(i)
- }
- return I
- }
- }
- Date.now || (Date.now = function() {
- return (new Date)
- .getTime()
- });
- var _ = " \n \f\r \u00a0\u1680\u180e\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u202f\u205f\u3000\u2028\u2029\ufeff";
- if (!String.prototype.trim || _.trim()) {
- _ = "[" + _ + "]";
- var D = new RegExp("^" + _ + _ + "*"),
- P = new RegExp(_ + _ + "*$");
- String.prototype.trim = function() {
- return String(this)
- .replace(D, "")
- .replace(P, "")
- }
- }
- var F = function(e) {
- if (e == null) throw new TypeError("can't convert " + e + " to object");
- return Object(e)
- }
-})
\ No newline at end of file
diff --git a/doc/swift-target.md b/doc/swift-target.md
index d023ee4f70..9f04349b9c 100644
--- a/doc/swift-target.md
+++ b/doc/swift-target.md
@@ -1,15 +1,10 @@
# ANTLR4 Language Target, Runtime for Swift
-## Requirements
-
-ANTLR 4.7.2 requires Swift 4.2. It works on Swift 4.2.1 also.
-
-ANTLR 4.7.1 requires Swift 4.0, and does not work on Swift 4.2. (The status of
-Swift 4.1 support is unknown.)
-
## Performance Note
-To use ANTLR4 Swift target in production environment, make sure to turn on compiler optimizations by following [these instructions](https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#build-configurations) if you use SwiftPM to build your project. If you are using Xcode to build your project, it's unlikely you will not use `release` build for production build.
+To use ANTLR4 Swift target in production environment, make sure to turn on compiler optimizations by following [these instructions](https://github.com/apple/swift-package-manager/blob/main/Documentation/Usage.md#setting-the-build-configuration) if you use SwiftPM to build your project.
+
+If you are using Xcode to build your project, it's unlikely you will not use `release` build for production build.
Conclusion is, you need to turn on `release` mode (which will have all the optimization pre configured for you) so the ANTLR4 Swift target can have reasonable parsing speed.
@@ -60,10 +55,7 @@ Note that even if you are otherwise using ANTLR from a binary distribution,
you should compile the ANTLR Swift runtime from source, because the Swift
language does not yet have a stable ABI.
-ANTLR uses Swift Package Manager to generate Xcode project files. Note that
-Swift Package Manager does not currently support iOS, watchOS, or tvOS, so
-if you wish to use those platforms, you will need to alter the project build
-settings manually as appropriate.
+ANTLR uses Swift Package Manager to generate Xcode project files.
#### Download source code for ANTLR
@@ -135,29 +127,12 @@ The runtime and generated grammar should now build correctly.
### Swift Package Manager Projects
-Since we cannot have a separate repository for Swift target (see issue [#1774](https://github.com/antlr/antlr4/issues/1774)),
-and Swift is currently not ABI stable. We currently support support SPM-based
-projects by creating temporary local repository.
+Add Antlr4 as a dependency to your `Package.swift` file. For more information, please see the [Swift Package Manager documentation](https://github.com/apple/swift-package-manager/tree/master/Documentation).
-For people using [Swift Package Manager](https://swift.org/package-manager/),
-the __boot.py__ script supports generating local repository that can be used
-as a dependency to your project. Simply run:
+```swift
+.package(url: "https://github.com/antlr/antlr4", from: "4.13.2")
```
-python boot.py --gen-spm-module
-```
-
-The prompt will show something like below:
-
-
-
-Put the SPM directive that contains the url to temporary repository to your
-project's Package.swift. And run `swift build` in your project.
-
-The project is generated in your system's `/tmp/` directory, if you find it
-inconvenient, consider copy that generated ANTLR repository to some place
-that won't be cleaned automatically and update `url` parameter in your
-`Package.swift` file.
## Swift access levels
diff --git a/doc/target-agnostic-grammars.md b/doc/target-agnostic-grammars.md
new file mode 100644
index 0000000000..62ee48231a
--- /dev/null
+++ b/doc/target-agnostic-grammars.md
@@ -0,0 +1,82 @@
+# Writing target-agnostic grammars
+
+Some grammars require
+[semantic predicates](https://github.com/antlr/antlr4/blob/dev/doc/predicates.md)
+to add context-sensitive parsing to what would generally be a context-free grammar.
+
+For example:
+* In Fortran90, [lines that begin with a 'C' in column 1
+are comments](https://github.com/antlr/grammars-v4/blob/43fbb16fec1d474d38a603cc6a6bcbe5edf07b1e/fortran/fortran90/slow/hw.f90#L1),
+which should be placed on a token stream other than the default.
+But, if the 'C' does not begin in
+column 1, then the input is invalid and should be flagged as so.
+ ```fortran
+ c Hello World.
+ c This is a syntax error because 'c' does not start in column 1
+ program hello
+ print *, 'Hello World!'
+ end
+ ```
+
+* In CSharp, two [greater-than signs](https://util.unicode.org/UnicodeJsps/character.jsp?a=003E)
+`'>>'` can either mean
+[a right shift expression](https://github.com/antlr/grammars-v4/blob/43fbb16fec1d474d38a603cc6a6bcbe5edf07b1e/csharp/examples/AllInOneNoPreprocessor.cs#L657C15-L657C17)
+or [part of a type declaration with templates](https://github.com/antlr/grammars-v4/blob/master/csharp/examples/AllInOneNoPreprocessor.cs#L463C33-L463C35).
+Since lexers in Antlr are not parser aware,
+the lexer must tokenize the two greater-than signs as two separate tokens.
+A semantic predicate should be added to disallow a space between the two greater-than signs in the context
+of an expression, but allowed in the context of a type declaration.
+ ```C#
+ class Foo {
+ void Func()
+ {
+ int x = 1000 > > 2; // syntax error if a space exists in the double greater-than sign
+ }
+ Dictionary > mapping; // nested template declaration, valid
+ }
+ ```
+
+Antlr does not have a general-purpose language for predicates. These must be
+written in the target language of the generated parser. The problem is that
+a grammar would need to be forked for each target desired, which adds to the
+burden of maintenance.
+
+However, it is possible to write the grammar such that forking is not required,
+using _target-agnostic format_.
+
+## Rules in writing target-agnostic grammars
+
+1) You will need to [split your grammar](https://github.com/antlr/antlr4/blob/dev/doc/grammars.md#grammar-structure)
+into separate lexer and parser grammars. Then, add `options { tokenVocab=...; }` to the parser grammar.
+2) Create target-specific source code files that contain methods in a base class for
+the parser or lexer grammar. In these source code files, write the code for the semantic
+predicate. For example, the files for the Cpp target would be `Python3LexerBase.{cpp,h}`, `Python3ParserBase.{cpp,h}`.
+3) In the grammar(s), add `options { superClass=... }`. This will
+[superclass the recognizer](https://github.com/antlr/antlr4/blob/dev/doc/options.md#superclass).
+For example, `options { superclass=Python3ParserBase; }`.
+4) In the grammar(s), write code to make a single
+call to the base-class method. The call should have a `this.` string
+before the name of the method, e.g., `OPEN_PAREN : '(' {this.openBrace();};`
+The action code must not reference Antlr attributes,
+variables, types, or have semi-colons as statement separators or
+control-flow statements of any kind.
+5) For some targets like Cpp and PHP, you may need to add code to include source
+code files so that the generated code compiles.
+For these, add a comment
+such as `// Insert here @header for lexer include.` or `// Insert here @header for parser include.`
+to the grammar, before the first rule.
+5) Add a Python script called "transformGrammar.py" that rewrites the grammar(s)
+with some target-specific code syntax.
+ a) For Cpp: replace `this.` strings with `this->`.
+ b) For PHP: replace `this.` strings with `$this->`.
+ c) For Python: replace `this.` strings with `self.`, `l.`, or `p.` depending on
+where the action or predicate is in the grammar.
+ d) For Cpp: replace `// Insert here @header for lexer include.` (or parser) with
+`@header::lexer {#include ...}`.
+ e) For PHP: replace `// Insert here @header for lexer include.` (or parser) with
+`@header::lexer {require ...}`.
+ e) Run `python transformGrammar.py *.g4` before generating the parser and lexer.
+
+## Examples of target-agnostic grammars
+* [fortran90](https://github.com/antlr/grammars-v4/tree/master/fortran/fortran90)
+* [csharp](https://github.com/antlr/grammars-v4/tree/master/csharp)
diff --git a/doc/targets.md b/doc/targets.md
index f2a63689ae..0c61b6f3a8 100644
--- a/doc/targets.md
+++ b/doc/targets.md
@@ -4,19 +4,21 @@ This page lists the available and upcoming ANTLR runtimes. Please note that you
* [Java](java-target.md). The [ANTLR v4 book](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) has a decent summary of the runtime library. We have added a useful XPath feature since the book was printed that lets you select bits of parse trees. See [Runtime API](http://www.antlr.org/api/Java/index.html) and [Getting Started with ANTLR v4](getting-started.md)
* [C#](csharp-target.md)
-* [Python](python-target.md) (2 and 3)
+* [Python](python-target.md) (3)
* [JavaScript](javascript-target.md)
+* [TypeScript](typescript-target.md)
* [Go](go-target.md)
* [C++](cpp-target.md)
* [Swift](swift-target.md)
* [PHP](php-target.md)
+* [Dart](dart-target.md)
* [Rust](rust-target.md) (Unstable)
## Target feature parity
New features generally appear in the Java target and then migrate to the other targets, but these other targets don't always get updated in the same overall tool release. This section tries to identify features added to Java that have not been added to the other targets.
-|Feature|Java|C♯|Python2|Python3|JavaScript|Go|C++|Swift|PHP
+|Feature|Java|C♯|Python3|JavaScript|Go|C++|Swift|PHP|Dart
|---|---|---|---|---|---|---|---|---|---|
|Ambiguous tree construction|4.5.1|-|-|-|-|-|-|-|-|
diff --git a/doc/tree-matching.md b/doc/tree-matching.md
index f4c6d278cc..b0ec83bb5a 100644
--- a/doc/tree-matching.md
+++ b/doc/tree-matching.md
@@ -52,15 +52,6 @@ m.setDelimiters("<<", ">>", "$"); // $ is the escape character
This would allow pattern `<> = <> ;$<< ick $>>` to be interpreted as elements: `ID`, ` = `, `expr`, and ` ;<< ick >>`.
-```java
-String xpath = "//blockStatement/*";
-String treePattern = "int = ;";
-ParseTreePattern p =
-parser.compileParseTreePattern(treePattern,
-JavaParser.RULE_localVariableDeclarationStatement);
-List matches = p.findAll(tree, xpath);
-```
-
### Pattern labels
The tree pattern matcher tracks the nodes in the tree at matches against the tags in a tree pattern. That way we can use the `get()` and `getAll()` methods to retrieve components of the matched subtree. For example, for pattern ``, `get("ID")` returns the node matched for that `ID`. If more than one node matched the specified token or rule tag, only the first match is returned. If there is no node associated with the label, this returns null.
diff --git a/doc/typescript-target.md b/doc/typescript-target.md
new file mode 100644
index 0000000000..af03575d03
--- /dev/null
+++ b/doc/typescript-target.md
@@ -0,0 +1,125 @@
+# TypeScript
+
+Antlr4 TypeScript runtime uses the JavaScript runtime and adds type files to it.
+This guarantees the same behaviour and performance across both target languages.
+Generated lexers, parsers, listeners and visitors are generated in TypeScript.
+
+The runtime is built using TypeScript v4.8.3, node 16.17 and webpack 5.66.
+It may work with older versions but they have not been tested and they will not be supported.
+
+
+## How to create a TypeScript lexer or parser?
+
+This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example:
+
+```bash
+$ antlr4 -Dlanguage=TypeScript MyGrammar.g4
+```
+
+For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md).
+
+## Where can I get the runtime?
+
+Once you've generated the lexer and/or parser code, you need to download the runtime from [npm](https://www.npmjs.com/package/antlr4).
+
+We will not document here how to refer to the runtime from your project, since this would differ a lot depending on your project type and IDE.
+
+## How do I get the runtime in my browser?
+
+The runtime is webpacked and sits in the dist folder. A .map file is also provided.
+
+## How do I run the generated lexer and/or parser?
+
+Let's suppose that your grammar is named, as above, "MyGrammar". Let's suppose this parser comprises a rule named "MyStartRule". The tool will have generated for you the following files:
+
+* MyGrammarLexer.ts
+* MyGrammarParser.ts
+* MyGrammarListener.ts (if you have not activated the -no-listener option)
+* MyGrammarVisitor.ts (if you have activated the -visitor option)
+
+There is no listener or visitor interface generated, instead the generated listener and visitor class methods are implemented using lambdas.
+
+Now a fully functioning script might look like the following:
+
+```typescript
+import { CharStream, CommonTokenStream } from 'antlr4';
+import MyGrammarLexer from './MyGrammarLexer';
+import MyGrammarParser from './MyGrammarParser';
+
+const input = "your text to parse here"
+const chars = new CharStream(input); // replace this with a FileStream as required
+const lexer = new MyGrammarLexer(chars);
+const tokens = new CommonTokenStream(lexer);
+const parser = new MyGrammarParser(tokens);
+const tree = parser.MyStartRule();
+
+```
+
+Tha above program will work. But it won't be useful unless you do one of the following:
+
+* you visit the parse tree using a custom listener
+* you visit the parse tree using a custom visitor
+* your grammar contains production code (like AntLR3)
+
+(please note that production code is target specific, so you can't have multi target grammars that include production code)
+
+## How do I create and run a visitor?
+
+You need to create a custom visitor and use it to visit the parse tree, as follows:
+```typescript
+
+import { ParserRuleContext } from 'antlr4';
+import MyGrammarVisitor from './MyGrammarVisitor';
+
+class CustomVisitor extends MyGrammarVisitor {
+
+ visitChildren(ctx: ParserRuleContext) {
+ if (!ctx) {
+ return;
+ }
+ if (ctx.children) {
+ return ctx.children.map(child => {
+ if (child.children && child.children.length != 0) {
+ return child.accept(this);
+ } else {
+ return child.getText();
+ }
+ });
+ }
+ }
+}
+
+tree.accept(new CustomVisitor());
+````
+
+## How do I create and run a custom listener?
+
+You need to create a custom listener and use it to visit the parse tree, as follows:
+
+```typescript
+
+import { ParseTreeWalker } from 'antlr4';
+import MyGrammarListener from './MyGrammarListener';
+
+class MyTreeWalker extends MyGrammarListener {
+
+ exitMyStartRule = (ctx: MyStartRuleContext) => {
+ console.log("In MyStartRule");
+ };
+
+}
+
+const walker = new MyTreeWalker();
+ParseTreeWalker.DEFAULT.walk(walker, tree);
+
+```
+
+## How do I integrate my parser with ACE editor?
+
+This specific task is described in this [dedicated page](ace-javascript-target.md).
+
+## How can I learn more about ANTLR?
+
+Further information can be found from "The definitive ANTLR 4 reference" book.
+
+The TypeScript implementation of ANTLR is based on the JavaScript implementation, which is as close as possible to the Java one, so you shouldn't find it difficult to adapt the book's examples to TypeScript.
diff --git a/doc/wildcard.md b/doc/wildcard.md
index f3d1c3a0b5..a597c649c9 100644
--- a/doc/wildcard.md
+++ b/doc/wildcard.md
@@ -61,7 +61,7 @@ END : '>>' ;
After crossing through a nongreedy subrule within a lexical rule, all decision-making from then on is "first match wins."
-For example, literal `ab` in rule right-hand side (grammar fragment) `.*? (’a’|’ab’)` is dead code and can never be matched. If the input is ab, the first alternative, ’a’, matches the first character and therefore succeeds. (’a’|’ab’) by itself on the right-hand side of a rule properly matches the second alternative for input ab. This quirk arises from a nongreedy design decision that’s too complicated to go into here.
+For example, literal `ab` in rule right-hand side (grammar fragment) `.*? ('a'|'ab')` is dead code and can never be matched. If the input is ab, the first alternative, 'a', matches the first character and therefore succeeds. ('a'|'ab') by itself on the right-hand side of a rule properly matches the second alternative for input ab. This quirk arises from a nongreedy design decision that’s too complicated to go into here.
-**Note:** You must sign the `contributors.txt` certificate of origin with your pull request if you've not done so before.
+**Note:** Each commit requires a "signature", which is simple as using `-s` (not
+`-S`) to the git commit command:
+
+```
+git commit -s -m 'This is my commit message'
+```
+
+Github's pull request process enforces the sig and gives instructions on how to
+fix any commits that lack the sig. See [Github DCO app](https://github.com/apps/dco)
+for more info.
diff --git a/LICENSE.txt b/LICENSE.txt
index 2042d1bda6..5d27694155 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -1,52 +1,28 @@
-[The "BSD 3-clause license"]
-Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
+Copyright (c) 2012-2022 The ANTLR Project. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
- 1. Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- 2. Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in the
- documentation and/or other materials provided with the distribution.
- 3. Neither the name of the copyright holder nor the names of its contributors
- may be used to endorse or promote products derived from this software
- without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
-IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
-OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
-IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
-INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
-NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
-THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-=====
-
-MIT License for codepointat.js from https://git.io/codepointat
-MIT License for fromcodepoint.js from https://git.io/vDW1m
-
-Copyright Mathias Bynens 
+
+The Go target now has its own dedicated repo:
+
+```bash
+$ go get github.com/antlr4-go/antlr
+```
+**Note**
+The dedicated Go repo is for `go get` and `import` only. Go runtime development is still performed in the main `antlr/antlr4` repo.
## Authors and major contributors
* [Terence Parr](http://www.cs.usfca.edu/~parrt/), parrt@cs.usfca.edu
ANTLR project lead and supreme dictator for life
[University of San Francisco](http://www.usfca.edu/)
-* [Sam Harwell](http://tunnelvisionlabs.com/) (Tool co-author, Java and C# target)
-* Eric Vergnaud (Javascript, Python2, Python3 targets and significant work on C# target)
+* [Sam Harwell](http://tunnelvisionlabs.com/) (Tool co-author, Java and original C# target)
+* [Eric Vergnaud](https://github.com/ericvergnaud) (Javascript, TypeScript, Python2, Python3 targets and maintenance of C# target)
* [Peter Boyer](https://github.com/pboyer) (Go target)
* [Mike Lischke](http://www.soft-gems.net/) (C++ completed target)
* Dan McLaughlin (C++ initial target)
@@ -23,6 +63,12 @@ ANTLR project lead and supreme dictator for life
* [Ewan Mellor](https://github.com/ewanmellor), [Hanzhou Shi](https://github.com/hanjoes) (Swift target merging)
* [Ben Hamilton](https://github.com/bhamiltoncx) (Full Unicode support in serialized ATN and all languages' runtimes for code points > U+FFFF)
* [Marcos Passos](https://github.com/marcospassos) (PHP target)
+* [Lingyu Li](https://github.com/lingyv-li) (Dart target)
+* [Ivan Kochurkin](https://github.com/KvanTTT) has made major contributions to overall quality, error handling, and Target performance.
+* [Justin King](https://github.com/jcking) has done a huge amount of work across multiple targets, but especially for C++.
+* [Ken Domino](https://github.com/kaby76) has a knack for finding bugs/issues and analysis; also a major contributor on the [grammars-v4 repo](https://github.com/antlr/grammars-v4).
+* [Jim Idle](https://github.com/jimidle) has contributed to previous versions of ANTLR and recently jumped back in to solve a major problem with the Go target.
+
## Useful information
@@ -31,7 +77,12 @@ ANTLR project lead and supreme dictator for life
* [Official site](http://www.antlr.org/)
* [Documentation](https://github.com/antlr/antlr4/blob/master/doc/index.md)
* [FAQ](https://github.com/antlr/antlr4/blob/master/doc/faq/index.md)
-* [ANTLR code generation targets](https://github.com/antlr/antlr4/blob/master/doc/targets.md)
+## Requirements
-And the result of testing the entire subdirectory:
+In order to perform the tests on all target languages, the following tools should be installed:
-
+* dotnet
+* Node.js
+* Python 3
+* Go
+* Swift
+* Clang (Linux, Mac) or MSBuild (Windows) for C++
+* Dart
+* PHP
-From `mvn`, on the commandline, you will see:
+To **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash
-$ cd antlr4
-$ mvn test
-...
--------------------------------------------------------
- T E S T S
--------------------------------------------------------
-Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
-dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
-Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
-dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
-Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
-Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
-Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
-dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
-antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
-...
-[INFO] ------------------------------------------------------------------------
-[INFO] Reactor Summary:
-[INFO]
-[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
-[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
-[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
-[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
-[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
-[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
-[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
-[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
-[INFO] ------------------------------------------------------------------------
-[INFO] BUILD SUCCESS
-[INFO] ------------------------------------------------------------------------
-[INFO] Total time: 06:53 min
-[INFO] Finished at: 2016-11-16T15:36:56-08:00
-[INFO] Final Memory: 44M/458M
-[INFO] ------------------------------------------------------------------------
+$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
+$ mvn install -DskipTests # make sure all artifacts are visible on this machine
```
-Note: That is actually result of running the much faster:
-
-```bash
-mvn -Dparallel=methods -DthreadCount=4 install
-```
+## Running the runtime tests
-## Running test subsets
+A single test rig is sufficient to test all targets against all descriptors using the [junit dynamic tests](https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests) mechanism.
+But it's often convenient to test just a single target or perhaps even just a single test within a single group of a single target.
+IntelliJ automatically generates a bunch of
+[Target runtime test rigs](../runtime-testsuite/test/org/antlr/v4/test/runtime) that allows developers such flexibility.
+For example, here are the Python3 test rigs in IntelliJ:
-*From the `runtime-testsuite` dir*
+
-### Run one test group across targets
+And the result of testing the entire subdirectory:
-```bash
-$ cd runtime-testsuite
-$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
-$ mvn -Dtest=TestParserExec test
--------------------------------------------------------
- T E S T S
--------------------------------------------------------
-Running org.antlr.v4.test.runtime.cpp.TestParserExec
-...
-Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
-Running org.antlr.v4.test.runtime.csharp.TestParserExec
-...
-```
+
-Or run all lexer related tests:
+All test are run in parallel both via maven and via IDE.
-```
-$ cd runtime-testsuite
-$ mvn -Dtest=Test*Lexer* test
--------------------------------------------------------
- T E S T S
--------------------------------------------------------
-Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
-...
-```
+In IntelliJ, it's very easy to go to source by right-clicking on any test and pressing `Jump to source` (F4).
-### Run all tests for a single target
+## Running test subsets
-```bash
-$ cd runtime-testsuite
-$ mvn -Dtest=java.* test
-...
-```
+From the `runtime-testsuite` dir
-Or run all lexer related tests in Java target only:
+### Run all tests for a single target
```bash
$ cd runtime-testsuite
-$ mvn -Dtest=java.*Lexer* test
-...
+$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
+$ mvn -Dtest='java.**' test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-Running org.antlr.v4.test.runtime.java.TestCompositeLexers
-Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
-Running org.antlr.v4.test.runtime.java.TestLexerErrors
-Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
-Running org.antlr.v4.test.runtime.java.TestLexerExec
-Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
-Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
-Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
-
-Results :
-
-Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
-```
-
-## Testing in parallel
-
-Use this to run tests in parallel:
-
-```bash
-$ export MAVEN_OPTS="-Xmx1G"
-$ mvn -Dparallel=methods -DthreadCount=4 test
+[INFO] Running org.antlr.v4.test.runtime.java.TestIntegerList
+[INFO] Running org.antlr.v4.test.runtime.java.JavaRuntimeTests
...
--------------------------------------------------------
- T E S T S
--------------------------------------------------------
-Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
+[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023 s - in org.antlr.v4.test.runtime.java.TestIntegerList
+[INFO] Tests run: 348, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.269 s - in org.antlr.v4.test.runtime.java.JavaRuntimeTests
...
```
-This can be combined with other `-D` above.
-
## Adding a runtime test
-To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:
+To add a new runtime test, first determine which [group (dir) of tests](../runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors) it belongs to.
+Then, add a new descriptor file implementation by filling in one of these (omitting unused sections):
-* [BaseParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseParserTestDescriptor.java); see example [APlus](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParserExecDescriptors.java#L7).
-* [BaseDiagnosticParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseDiagnosticParserTestDescriptor) if you want to test parser diagnostic output; see [example output](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/FullContextParsingDescriptors.java#L16).
-* [BaseCompositeParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeParserTestDescriptor.java); see example [BringInLiteralsFromDelegate](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeParsersDescriptors.java#L11)
-* [BaseLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseLexerTestDescriptor.java); see example [ActionPlacement](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/LexerExecDescriptors.java#L12).
-* [BaseCompositeLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeLexerTestDescriptor.java); see example [LexerDelegatorInvokesDelegateRule](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeLexersDescriptors.java#L11)
+```
+[notes]
+[type]
-Each descriptor object describes the following mandatory elements for the test:
+[grammar]
- * the test type
- * the grammar
- * the start rule
- * the input text to parse or lex
- * the expected output
- * the expected errors
+[slaveGrammar]
-Your best bet is to find a similar test in the appropriate group and then copy and paste the descriptor object, creating a new nested class within the test group class. Modify the field definitions to suit your new problem.
+[start]
-If you need to create a whole new group of tests, it requires a new descriptor class; call it `XDescriptors`. Then, in each [target subdirectory](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime), you need to create a new test rig `TestX.java` file:
+[input]
-```java
-package org.antlr.v4.test.runtime.java;
+[output]
-import org.antlr.v4.test.runtime.BaseRuntimeTest;
-import org.antlr.v4.test.runtime.RuntimeTestDescriptor;
-import org.antlr.v4.test.runtime.descriptors.ListenersDescriptors;
-import org.junit.runner.RunWith;
-import org.junit.runners.Parameterized;
+[errors]
-@RunWith(Parameterized.class)
-public class TestX extends BaseRuntimeTest {
- public TestX(RuntimeTestDescriptor descriptor) {
- super(descriptor,new Base
+
+### Generating parser code
+
+The previous section used a built-in ANTLR interpreter but typically you will ask ANTLR to generate code in the language used by your project (there are about 10 languages to choose from as of 4.11). Here's how to generate Java code from a grammar:
+
+```bash
+$ antlr4 Expr.g4
+$ ls Expr*.java
+ExprBaseListener.java ExprLexer.java ExprListener.java ExprParser.java
+```
+
+And, here's how to generate C++ code from the same grammar:
+
+```bash
+$ antlr4 -Dlanguage=Cpp Expr.g4
+$ ls Expr*.cpp Expr*.h
+ExprBaseListener.cpp ExprLexer.cpp ExprListener.cpp ExprParser.cpp
+ExprBaseListener.h ExprLexer.h ExprListener.h ExprParser.h
+```
## Installation
-ANTLR is really two things: a tool that translates your grammar to a parser/lexer in Java (or other target language) and the runtime needed by the generated parsers/lexers. Even if you are using the ANTLR Intellij plug-in or ANTLRWorks to run the ANTLR tool, the generated code will still need the runtime library.
+ANTLR is really two things: a tool written in Java that translates your grammar to a parser/lexer in Java (or other target language) and the runtime library needed by the generated parsers/lexers. Even if you are using the ANTLR Intellij plug-in or ANTLRWorks to run the ANTLR tool, the generated code will still need the runtime library.
-The first thing you should do is probably download and install a development tool plug-in. Even if you only use such tools for editing, they are great. Then, follow the instructions below to get the runtime environment available to your system to run generated parsers/lexers. In what follows, I talk about antlr-4.7.1-complete.jar, which has the tool and the runtime and any other support libraries (e.g., ANTLR v4 is written in v3).
+The first thing you should do is probably download and install a development tool plug-in. Even if you only use such tools for editing, they are great. Then, follow the instructions below to get the runtime environment available to your system to run generated parsers/lexers. In what follows, I talk about antlr-4.13.2-complete.jar, which has the tool and the runtime and any other support libraries (e.g., ANTLR v4 is written in v3).
-If you are going to integrate ANTLR into your existing build system using mvn, ant, or want to get ANTLR into your IDE such as eclipse or intellij, see Integrating ANTLR into Development Systems.
+If you are going to integrate ANTLR into your existing build system using mvn, ant, or want to get ANTLR into your IDE such as eclipse or intellij, see [Integrating ANTLR into Development Systems](https://github.com/antlr/antlr4/blob/master/doc/IDEs.md).
### UNIX
-0. Install Java (version 1.6 or higher)
+0. Install Java (version 11 or higher)
1. Download
```
$ cd /usr/local/lib
-$ curl -O https://www.antlr.org/download/antlr-4.7.1-complete.jar
+$ curl -O https://www.antlr.org/download/antlr-4.13.2-complete.jar
```
Or just download in browser from website:
[https://www.antlr.org/download.html](https://www.antlr.org/download.html)
and put it somewhere rational like `/usr/local/lib`.
-2. Add `antlr-4.7.1-complete.jar` to your `CLASSPATH`:
+if you are using lower version jdk, just download from [website download](https://github.com/antlr/website-antlr4/tree/gh-pages/download) for previous version, and antlr version before 4.13.2 support jdk 1.8
+
+2. Add `antlr-4.13.2-complete.jar` to your `CLASSPATH`:
```
-$ export CLASSPATH=".:/usr/local/lib/antlr-4.7.1-complete.jar:$CLASSPATH"
+$ export CLASSPATH=".:/usr/local/lib/antlr-4.13.2-complete.jar:$CLASSPATH"
```
It's also a good idea to put this in your `.bash_profile` or whatever your startup script is.
3. Create aliases for the ANTLR Tool, and `TestRig`.
```
-$ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.7.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
-$ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.7.1-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig'
+$ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.13.2-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
+$ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.13.2-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig'
```
### WINDOWS
(*Thanks to Graham Wideman*)
-0. Install Java (version 1.6 or higher)
-1. Download antlr-4.7.1-complete.jar (or whatever version) from [https://www.antlr.org/download/](https://www.antlr.org/download/)
+0. Install Java (version 1.7 or higher)
+1. Download antlr-4.13.2-complete.jar (or whatever version) from [https://www.antlr.org/download.html](https://www.antlr.org/download.html)
Save to your directory for 3rd party Java libraries, say `C:\Javalib`
-2. Add `antlr-4.7.1-complete.jar` to CLASSPATH, either:
+2. Add `antlr-4.13.2-complete.jar` to CLASSPATH, either:
* Permanently: Using System Properties dialog > Environment variables > Create or append to `CLASSPATH` variable
* Temporarily, at command line:
```
-SET CLASSPATH=.;C:\Javalib\antlr-4.7.1-complete.jar;%CLASSPATH%
+SET CLASSPATH=.;C:\Javalib\antlr-4.13.2-complete.jar;%CLASSPATH%
```
3. Create short convenient commands for the ANTLR Tool, and TestRig, using batch files or doskey commands:
* Batch files (in directory in system PATH) antlr4.bat and grun.bat
@@ -71,7 +197,7 @@ Either launch org.antlr.v4.Tool directly:
```
$ java org.antlr.v4.Tool
-ANTLR Parser Generator Version 4.7.1
+ANTLR Parser Generator Version 4.13.2
-o ___ specify output directory where all output is generated
-lib ___ specify location of .tokens files
...
@@ -80,8 +206,8 @@ ANTLR Parser Generator Version 4.7.1
or use -jar option on java:
```
-$ java -jar /usr/local/lib/antlr-4.7.1-complete.jar
-ANTLR Parser Generator Version 4.7.1
+$ java -jar /usr/local/lib/antlr-4.13.2-complete.jar
+ANTLR Parser Generator Version 4.13.2
-o ___ specify output directory where all output is generated
-lib ___ specify location of .tokens files
...
@@ -133,7 +259,12 @@ That pops up a dialog box showing that rule `r` matched keyword `hello` followed
The book has lots and lots of examples that should be useful too. You can download them here for free:
-[http://pragprog.com/titles/tpantlr2/source_code](http://pragprog.com/titles/tpantlr2/source_code)
+[ANTLR reference book examples in Java](https://media.pragprog.com/titles/tpantlr2/code/tpantlr2-code.zip)
-
-Put the SPM directive that contains the url to temporary repository to your
-project's Package.swift. And run `swift build` in your project.
-
-The project is generated in your system's `/tmp/` directory, if you find it
-inconvenient, consider copy that generated ANTLR repository to some place
-that won't be cleaned automatically and update `url` parameter in your
-`Package.swift` file.
## Swift access levels
diff --git a/doc/target-agnostic-grammars.md b/doc/target-agnostic-grammars.md
new file mode 100644
index 0000000000..62ee48231a
--- /dev/null
+++ b/doc/target-agnostic-grammars.md
@@ -0,0 +1,82 @@
+# Writing target-agnostic grammars
+
+Some grammars require
+[semantic predicates](https://github.com/antlr/antlr4/blob/dev/doc/predicates.md)
+to add context-sensitive parsing to what would generally be a context-free grammar.
+
+For example:
+* In Fortran90, [lines that begin with a 'C' in column 1
+are comments](https://github.com/antlr/grammars-v4/blob/43fbb16fec1d474d38a603cc6a6bcbe5edf07b1e/fortran/fortran90/slow/hw.f90#L1),
+which should be placed on a token stream other than the default.
+But, if the 'C' does not begin in
+column 1, then the input is invalid and should be flagged as so.
+ ```fortran
+ c Hello World.
+ c This is a syntax error because 'c' does not start in column 1
+ program hello
+ print *, 'Hello World!'
+ end
+ ```
+
+* In CSharp, two [greater-than signs](https://util.unicode.org/UnicodeJsps/character.jsp?a=003E)
+`'>>'` can either mean
+[a right shift expression](https://github.com/antlr/grammars-v4/blob/43fbb16fec1d474d38a603cc6a6bcbe5edf07b1e/csharp/examples/AllInOneNoPreprocessor.cs#L657C15-L657C17)
+or [part of a type declaration with templates](https://github.com/antlr/grammars-v4/blob/master/csharp/examples/AllInOneNoPreprocessor.cs#L463C33-L463C35).
+Since lexers in Antlr are not parser aware,
+the lexer must tokenize the two greater-than signs as two separate tokens.
+A semantic predicate should be added to disallow a space between the two greater-than signs in the context
+of an expression, but allowed in the context of a type declaration.
+ ```C#
+ class Foo {
+ void Func()
+ {
+ int x = 1000 > > 2; // syntax error if a space exists in the double greater-than sign
+ }
+ Dictionary
\ No newline at end of file
+