Skip to content

How to build ANTLR itself

Jump to bottom Edit New page
Terence Parr edited this page Aug 20, 2015 · 25 revisions

out of date. use maven

See also Deploying ANTLR mvn artifacts.

Most programmers do not need the information on this page because they will simply download the appropriate jar(s) or use ANTLR through maven (via ANTLR's antlr4-maven-plugin). If you would like to fork the project and fix bugs or tweak the runtime code generation, then you will almost certainly need to build ANTLR itself. There are two components:

  1. the tool that compiles grammars down into parsers and lexers in one of the target languages
  2. the runtime used by those generated parsers and lexers.

As of 4.4, we use a Python script in the main directory called bild.py, which will bootstrap by pulling in the bilder.py library. You don't need to know much about what's inside, but the code is reasonably easy to read.

I will assume that the root directory is /tmp for the purposes of explaining how to build ANTLR in this document.

Get the source

The first step is to get the Java source code from the ANTLR 4 repository at github. You can download the repository from github, but the easiest thing to do is simply clone the repository on your local disk:

$ cd /tmp
/tmp $ git clone git@github.com:antlr/antlr4.git
Cloning into 'antlr4'...
remote: Counting objects: 43273, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 43273 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (43273/43273), 18.76 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (22419/22419), done.
Checking connectivity... done.

Really though you'll need all the targets to build jars etc...

cd /tmp
git clone git@github.com:antlr/antlr4.git
git clone git@github.com:antlr/antlr4-csharp.git
git clone git@github.com:antlr/antlr4-javascript.git
git clone git@github.com:antlr/antlr4-python2.git
git clone git@github.com:antlr/antlr4-python3.git

Freshen repos

cd /tmp
cd antlr4; git pull origin master; cd ..
cd antlr4-csharp; git pull origin master; cd ..
cd antlr4-javascript; git pull origin master; cd ..
cd antlr4-python2; git pull origin master; cd ..
cd antlr4-python3; git pull origin master; cd ..

also useful:

cd /tmp
cd antlr4; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-csharp; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-javascript; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-python2; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-python3; git commit -a -m 'msg'; git push origin master; cd ..

Compiling

First, make sure you have java installed. Type java on command line and it should launch. If not, install Java.

Now, let's make sure everything is clean just out of habit:

/tmp $ cd antlr4
/tmp/antlr4 $ ./bild.py clean
bootstrapping; downloading bilder.py
target clean
bild succeeded

Compiling the source code is easy (but note that compile will fail until you download the target repos—see below):

/tmp/antlr4 $ ./bild.py compile
target compile
require parsers
build compile
bild succeeded

Take a look in the bild.log file that is always produced if you'd like to see precisely what commands were executed and their output:

/tmp/antlr4 $ head bild.log
[01/13/15 18:56:16 <module> ./bild.py:39 bilder.py:107] platform=darwin
[01/13/15 18:56:16 <module> ./bild.py:39 bilder.py:108] jdk={'1.6': '/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home', '1.7': '/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home', '1.8': '/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home'}
[01/13/15 18:56:16] require parsers
[01/13/15 18:56:16 require ./bild.py:74 bilder.py:370] require parsers
[01/13/15 18:56:16 antlr3 ./bild.py:67 bilder.py:402] java -cp /Users/parrt/.bild/jars/antlr-3.5.1-complete.jar org.antlr.Tool -o /private/tmp/antlr4/gen3/org/antlr/v4/parse /private/tmp/antlr4/tool/src/org/antlr/v4/parse/GrammarTreeVisitor.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/BlockSetTransformer.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/LeftRecursiveRuleWalker.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ANTLRLexer.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ANTLRParser.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ATNBuilder.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ActionSplitter.g
[01/13/15 18:56:19 antlr3 ./bild.py:69 bilder.py:402] java -cp /Users/parrt/.bild/jars/antlr-3.5.1-complete.jar org.antlr.Tool -o /private/tmp/antlr4/gen3/org/antlr/v4/codegen -lib /private/tmp/antlr4/gen3/org/antlr/v4/parse /private/tmp/antlr4/tool/src/org/antlr/v4/codegen/SourceGenTriggers.g
[01/13/15 18:56:20 antlr4 ./bild.py:71 bilder.py:423] java -cp /Users/parrt/.bild/jars/antlr-4.4-complete.jar org.antlr.v4.Tool -o /private/tmp/antlr4/gen4/org/antlr/v4/runtime/tree/xpath -package org.antlr.v4.runtime.tree.xpath /private/tmp/antlr4/runtime/Java/src/org/antlr/v4/runtime/tree/xpath/XPathLexer.g4
[01/13/15 18:56:20] build compile
[01/13/15 18:56:20 require ./bild.py:74 bilder.py:378] build compile
[01/13/15 18:56:20 javac ./bild.py:81 bilder.py:452] javac -version
...

BTW, if you get an error about Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar see stackoverflow solution. Just:

$ unset JAVA_TOOL_OPTIONS

You will also notice that the build script has pulled in the necessary jars, sort of like maven would:

$ ls ~/.bild/jars/
ST-4.0.8.jar			antlr-4.5-SNAPSHOT-complete.jar	bild-junit.jar
antlr-3.5.1-complete.jar	antlr-4.5-complete.jar		hamcrest-core-1.3.jar
antlr-3.5.1-runtime.jar		antlr-4.5-rc-1-complete.jar	junit-4.11.jar
antlr-4.4-complete.jar		antlr4-4.5-complete.jar

Making tool and runtime jars

Because the build script expects all targets (the usual case), you must pull all of those target repositories in as well. We have kept them as separate repositories for organizational purposes and not because they are separate projects.

Here are the github repositories for the various targets (Java is contained within the main antlr4 tool):

You can run the following commands:

cd /tmp
git clone https://github.com/antlr/antlr4.git
git clone git@github.com:antlr/antlr4-csharp.git
git clone git@github.com:antlr/antlr4-javascript.git
git clone git@github.com:antlr/antlr4-python2.git
git clone git@github.com:antlr/antlr4-python3.git

Then go back into the antlr4 where the build script is and try to make a jar to see if everything is working.

/tmp $ cd antlr4
/tmp/antlr4 $ grep VERSION bild.py | head -2
BOOTSTRAP_VERSION = "4.4"
VERSION = "4.5-SNAPSHOT"
/tmp/antlr4 $ ./bild.py mkjar
target mkjar
require compile
require parsers
build compile
build mkjar_complete
Generated dist/antlr4-4.5-SNAPSHOT-complete.jar
require compile
Generated dist/antlr4-4.5-SNAPSHOT-complete.jar
Generated dist/antlr4-4.5-SNAPSHOT.jar
bild succeeded
/tmp/antlr4 $ ls dist
antlr4-4.5-SNAPSHOT-complete.jar	antlr4-4.5-SNAPSHOT.jar

You'll see that it builds the complete jar twice. ANTLR 4.5 has an ANTLR 4 grammar in it which must be built with the previous version because we cannot assume that the current version is available. Once we have built a valid ANTLR 4.5 using, say, 4.4, then we rebuild it using itself.

Testing tool and targets

In order to perform the tests, make sure that you have all of the targets per the previous section. Also you will need mono and `nodejs; on OS X:

$ brew install mono
$ brew install node

Then, launch the tests of all targets with:

./bild.py tests

This should peg all of your CPUs as it tries to launch the tests in parallel. Note that we have seen some tests fail when the C# target tests are run in parallel. They work fine in serial always (e.g., from your development environment).

You should see output like

...
Testing Python3 ...
org.antlr.v4.test.rt.py3.TestCompositeLexers: 2 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSemPredEvalLexer: 7 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParseTrees: 8 tests, 0 failures
org.antlr.v4.test.rt.py3.TestListeners: 7 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLexerErrors: 12 tests, 0 failures
org.antlr.v4.test.rt.py3.TestCompositeParsers: 15 tests, 0 failures
org.antlr.v4.test.rt.py3.TestFullContextParsing: 15 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSets: 23 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParserErrors: 26 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParserExec: 32 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSemPredEvalParser: 26 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLeftRecursion: 95 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLexerExec: 38 tests, 0 failures
Python3 tests complete
Testing JavaScript ...
org.antlr.v4.test.rt.js.node.TestBitSetWordSize: 1 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestCompositeLexers: 2 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSemPredEvalLexer: 7 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestListeners: 7 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParseTrees: 8 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLexerErrors: 12 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestCompositeParsers: 15 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestFullContextParsing: 15 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSets: 23 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParserErrors: 26 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSemPredEvalParser: 26 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParserExec: 32 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLexerExec: 38 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLeftRecursion: 95 tests, 0 failures
JavaScript tests complete
bild succeeded

Building ANTLR in Intellij

I build ANTLR during development in Intellij directly. The necessary project files are included in the repository in directory .idea and files:

$ cd /tmp/antlr4
$ find . -name '*.iml'
./antlr4.iml
./runtime/Java/runtime.iml
./tool/tool.iml

Making ANTLR runtime jar OSGi ready

Without being OSGi compatible, the ANTLR runtime cannot be used within eclipse. That's annoying. To fix this, we have a pull request that explains what the issue is with OSGi and describes how all we need to do is change the manifest file in the jar.

First, download bnd tool and make an alias for convenience:

alias bnd='java -jar /usr/local/lib/biz.aQute.bnd-3.0.0.jar'

Our regular mkjar bild target generates the following manifest:

Manifest-Version: 1.0
Implementation-Vendor: ANTLR
Implementation-Title: ANTLR 4 Runtime
Implementation-Version: 4.5
Implementation-Vendor-Id: org.antlr
Build-Jdk: 1.6
Built-By: parrt
Created-By: http://www.bildtool.org

Interesting. This prints exactly the existing jar manifest looks like:

$ bnd antlr4-4.5.jar 
[MANIFEST antlr4-4.5]
Build-Jdk                                1.6                                     
Built-By                                 parrt                                   
Created-By                               http://www.bildtool.org                 
Implementation-Title                     ANTLR 4 Runtime                         
Implementation-Vendor                    ANTLR                                   
Implementation-Vendor-Id                 org.antlr                               
Implementation-Version                   4.5                                     
Manifest-Version                         1.0

Well that was easy, I ran the following to get the proper jar.

cd dist
bnd wrap --output antlr4-4.5-osgi.jar antlr4-4.5.jar

The manifest file it generates is:

Manifest-Version: 1.0
Bnd-LastModified: 1421371533578
Build-Jdk: 1.6
Built-By: parrt
Bundle-ManifestVersion: 2
Bundle-Name: dist
Bundle-SymbolicName: dist
Bundle-Version: 0
Created-By: 1.7.0_25 (Oracle Corporation)
Export-Package: org.abego.treelayout,org.abego.treelayout.internal.util,
 org.abego.treelayout.internal.util.java.lang,org.abego.treelayout.inter
 nal.util.java.lang.string,org.abego.treelayout.internal.util.java.util,
 org.abego.treelayout.util;uses:="org.abego.treelayout",org.antlr.v4.run
 time;uses:="javax.print,javax.swing,org.antlr.v4.runtime.atn,org.antlr.
 v4.runtime.dfa,org.antlr.v4.runtime.misc,org.antlr.v4.runtime.tree,org.
 antlr.v4.runtime.tree.pattern",org.antlr.v4.runtime.atn;uses:="org.antl
 r.v4.runtime,org.antlr.v4.runtime.dfa,org.antlr.v4.runtime.misc",org.an
 tlr.v4.runtime.dfa;uses:="org.antlr.v4.runtime,org.antlr.v4.runtime.atn
 ",org.antlr.v4.runtime.misc;uses:="javax.annotation.processing,javax.la
 ng.model,javax.lang.model.element,javax.print,javax.swing,org.antlr.v4.
 runtime",org.antlr.v4.runtime.tree;uses:="org.antlr.v4.runtime,org.antl
 r.v4.runtime.misc",org.antlr.v4.runtime.tree.gui;uses:="javax.print,jav
 ax.swing,org.abego.treelayout,org.antlr.v4.runtime.tree",org.antlr.v4.r
 untime.tree.pattern;uses:="org.antlr.v4.runtime,org.antlr.v4.runtime.mi
 sc,org.antlr.v4.runtime.tree",org.antlr.v4.runtime.tree.xpath;uses:="or
 g.antlr.v4.runtime,org.antlr.v4.runtime.atn,org.antlr.v4.runtime.dfa,or
 g.antlr.v4.runtime.tree"
Implementation-Title: ANTLR 4 Runtime
Implementation-Vendor: ANTLR
Implementation-Vendor-Id: org.antlr
Implementation-Version: 4.5
Import-Package: javax.annotation.processing;resolution:=optional,javax.i
 mageio;resolution:=optional,javax.lang.model;resolution:=optional,javax
 .lang.model.element;resolution:=optional,javax.lang.model.type;resoluti
 on:=optional,javax.lang.model.util;resolution:=optional,javax.print;res
 olution:=optional,javax.print.attribute;resolution:=optional,javax.swin
 g;resolution:=optional,javax.swing.border;resolution:=optional,javax.sw
 ing.event;resolution:=optional,javax.swing.filechooser;resolution:=opti
 onal,javax.swing.tree;resolution:=optional,javax.tools;resolution:=opti
 onal
Originally-Created-By: http://www.bildtool.org
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.6))"
Tool: Bnd-3.0.0.201501151451

Use the following to test the updated jar:

bnd print --verify antlr4-4.5-osgi.jar

Ok, added to mkjar target so happens automatically now. It downloads 2.4 bnd (not 3.0) and wraps.

Editing the ANTLR github wiki

If you are a kind soul and would like to fix some of the documentation associated with building and managing handler, such as this page, you can fork antlr/antlr4 at github and then clone from YOUR_ID:

$ cd /tmp
$ git clone git@github.com:YOUR_ID/antlr4.wiki.git
Add a custom footer
Add a custom sidebar
Clone this wiki locally