HiveVM - Parser Generator

A parser generator is a tool that reads a grammar specification and converts it into a program that recognizes matches to that grammar.

HiveVM Compiler-Compiler (HiveVM CC) started as a fork of JavaCC 7.0.13, but it is no longer grammar-compatible with JavaCC. It keeps JavaCC's proven conceptual model — LL(k) recursive-descent parsing, a token manager with lexical states, syntactic and semantic lookahead, tree building — while replacing the surface syntax and the tooling around it:

• A simplified, EBNF-like syntax. Productions read name = expansion ; instead of ReturnType name() : {} { … }. There is no PARSER_BEGIN/PARSER_END wrapper embedding a Java class in the grammar; a grammar Name; line plus an options { … } block opens the file, and semantic actions are explicitly marked host-language islands, <? … ?>.
• One grammar, one step. Tree building is expressed in the same grammar with #Node annotations. The separate JJTree pre-processing pass is gone — there is no longer a two-step .jjt → .jj → parser pipeline, just a single grammar compiled in one step.
• Multiple targets. The same grammar can be emitted as Java, C++, or Rust.
• A more maintainable codebase, driven by a Gradle plugin rather than a CLI.

Because the syntax was redesigned, JavaCC grammars must be migrated, not dropped in; compatibility is at the concept and feature level, not the source level.

All you need to run a generated Java parser is a Java Runtime Environment (JRE) — generated parsers carry no HiveVM CC runtime dependency.

This repository is set up for agentic coding inside a ready-to-use Dev Container: coding agents work against explicit written rules so intent and reasoning stay reviewable.

For agent instructions, see AGENTS.md — the single source of truth for all coding agents.

Getting Started

1. Open the repository in VS Code and choose Reopen in Container — the Dev Container and preconfigured agent extensions build automatically.
2. Authenticate your coding agent inside the container (for Claude Code: claude login).
3. Start working with the agent — the rules it follows live in AGENTS.md.

Build, Test & Run

This is the single source for build/test/run commands — both humans and agents rely on it (AGENTS.md links here).

• Build: ./gradlew build
• Test: ./gradlew test
• Run: apply the org.hivevm.cc Gradle plugin and configure a parserProject (see Configure Settings below).

Configure Settings

Apply the org.hivevm.cc Gradle plugin and describe the generation units in a parserProject block. Each task produces one parser from one grammar; target selects the output language (java, cpp, or rust) and may be set project-wide or per task. The generateParser task then runs them.

Because tree building is part of the grammar, a grammar that builds a tree is not a separate kind of generation unit — it is an ordinary task whose grammar happens to use #Node annotations. Use treeNodes only to name node classes you supply yourself instead of having them generated.

plugins {
  id "org.hivevm.cc" version "1.0.10"
}

parserProject {
  target = 'java'                 // default output language: java | cpp | rust
  output = 'src/main/generated'   // default output directory

  task {
    name = 'parser'
    file = 'src/main/resources/JavaCC.jj'
  }

  task {
    name      = 'tree'
    file      = 'src/main/resources/JJTree.jj'
    treeNodes = [ 'BNF', 'NodeDescriptor' ]   // hand-written node classes
  }
}

Features

• HiveVM CC generates top-down (recursive descent) parsers as opposed to bottom-up parsers generated by YACC-like tools. This allows the use of more general grammars, although left-recursion is disallowed. Top-down parsers have a number of other advantages (besides more general grammars) such as being easier to debug, having the ability to parse to any non-terminal in the grammar, and also having the ability to pass values (attributes) both up and down the parse tree during parsing.

• By default, HiveVM CC generates an LL(1) parser. However, there may be portions of grammar that are not LL(1). HiveVM CC offers the capabilities of syntactic and semantic lookahead to resolve shift-shift ambiguities locally at these points. For example, the parser is LL(k) only at such points, but remains LL(1) everywhere else for better performance. Shift-reduce and reduce-reduce conflicts are not an issue for top-down parsers.

• HiveVM CC generates parsers that are 100% pure Java, so there is no runtime dependency on HiveVM CC and no special porting effort required to run on different machine platforms.

• HiveVM CC allows extended BNF specifications - such as (A)*, (A)+ etc - within the lexical and the grammar specifications. Extended BNF relieves the need for left-recursion to some extent. In fact, extended BNF is often easier to read as in A ::= y(x)* versus A ::= Ax|y.

• The lexical specifications (such as regular expressions, strings) and the grammar specifications (the BNF) are both written together in the same file. It makes grammars easier to read since it is possible to use regular expressions inline in the grammar specification, and also easier to maintain.

• The lexical analyzer of HiveVM CC can handle full Unicode input, and lexical specifications may also include any Unicode character. This facilitates descriptions of language elements such as Java identifiers that allow certain Unicode characters (that are not ASCII), but not others.

• HiveVM CC offers Lex-like lexical state and lexical action capabilities. Specific aspects in HiveVM CC that are superior to other tools are the first class status it offers concepts such as TOKEN, MORE, SKIP and state changes. This allows cleaner specifications as well as better error and warning messages from HiveVM CC.

• Tokens that are defined as special tokens in the lexical specification are ignored during parsing, but these tokens are available for processing by the tools. A useful application of this is in the processing of comments.

• Lexical specifications can define tokens to be case-insensitive per token block, with the [IGNORE_CASE] modifier (TOKEN [IGNORE_CASE] = …). Note that IGNORE_CASE is a reserved word and so cannot be used as a key inside options { … }.

• Tree building is part of the grammar itself: a production or an expansion is annotated with #Node, and the tree-node classes and visitor are generated alongside the parser. Unlike JavaCC — where JJTree is a separate pre-processor that rewrites a .jjt into a .jj before the parser generator runs — HiveVM CC needs no second step and no intermediate grammar.

• HiveVM CC also includes JJDoc, a tool that converts grammar files to documentation files, optionally in HTML.

• HiveVM CC offers many options to customize its behavior and the behavior of the generated parsers. Examples of such options are the kinds of Unicode processing to perform on the input stream, the number of tokens of ambiguity checking to perform etc.

• HiveVM CC error reporting is among the best in parser generators. HiveVM CC generated parsers are able to clearly point out the location of parse errors with complete diagnostic information.

• Using options DEBUG_PARSER, DEBUG_LOOKAHEAD, and DEBUG_TOKEN_MANAGER, users can get in-depth analysis of the parsing and the token processing steps.

• The HiveVM CC release includes a wide range of examples including Java and HTML grammars. The examples, along with their documentation, are a great way to get acquainted with HiveVM CC.

Example

This example recognizes matching braces followed by zero or more line terminators and then an end of file.

Examples of legal strings in this grammar are:

{}, {{{{{}}}}} // ... etc

Examples of illegal strings are:

{}{}, }{}}, { }, {x} // ... etc

Grammar

The productions and the token definitions live in the same file. Note that no Java class is embedded in the grammar: grammar Example; declares the grammar's name and options { … } configures generation. For the Java target the generated classes are always Parser, Lexer and ParserConstants, in the package given by JAVA_PACKAGE.

grammar Example;

options {
  JAVA_PACKAGE: "org.example"
}

/** Root production. */
Input =
  MatchedBraces() ( <EOL> )* <EOF>
;

/** Brace matching production. */
MatchedBraces =
  < LBRACE > [ MatchedBraces() ] < RBRACE >
;

SKIP =
  " "
| "\t"
;

TOKEN =
  < LBRACE: "{" >
| < RBRACE: "}" >
| < EOL: "\n" | "\r" | "\r\n" >
;

The same grammar written for JavaCC would need a PARSER_BEGIN(Example) … PARSER_END(Example) wrapper around a full Java class, void Input() : {} { … } productions, and { … } action blocks. None of that is accepted here — see ADR-0008.

Output

$ java Example
{{}}<return>

$ java Example
{x<return>
Lexical error at line 1, column 2.  Encountered: "x"
TokenMgrError: Lexical error at line 1, column 2.  Encountered: "x" (120), after : ""
        at ExampleTokenManager.getNextToken(ExampleTokenManager.java:146)
        at Example.getToken(Example.java:140)
        at Example.MatchedBraces(Example.java:51)
        at Example.Input(Example.java:10)
        at Example.main(Example.java:6)

$ java Example
{}}<return>
ParseException: Encountered "}" at line 1, column 3.
Was expecting one of:
    <EOF>
    "\n" ...
    "\r" ...
        at Example.generateParseException(Example.java:184)
        at Example.jj_consume_token(Example.java:126)
        at Example.Input(Example.java:32)
        at Example.main(Example.java:6)

Documentation

• Tutorials — docs/tutorials/: writing grammars for HiveVM CC — the token manager, lookahead, character input, error handling, lexer tips, and worked examples.
• Specification — docs/SPECIFICATION.md: problem, goals, and vocabulary.
• Architecture Decision Records — docs/adr/: the binding design decisions.

Dev Container

The environment is defined entirely in .devcontainer/devcontainer.json: it starts from a prebuilt base image and layers Dev Container Features and VS Code extensions on top — no Dockerfile or Compose file required. Customise the environment by adding Features, switching the base image, or adding extensions.

Coding Agents

This Dev Container preinstalls the Claude Code and Mistral Vibe VS Code extensions (see .devcontainer/devcontainer.json); other agents (OpenAI Codex, Cursor, OpenCode, GitHub Copilot) work too once you add them. Authenticate your agent inside the container (for Claude Code: claude login).

The rules every agent follows live in AGENTS.md — the single source of truth.

Contributing

See CONTRIBUTING.md for the workflow and CODE_OF_CONDUCT.md for the community standards we expect of everyone taking part. Security issues: please follow SECURITY.md instead of opening a public issue.

License

HiveVM CC is an open source project released under the BSD 3-Clause License. The JavaCC project was originally developed at Sun Microsystems Inc. by Sreeni Viswanadha and Sriram Sankar.