Extending Hammer

This page covers extending Hammer itself: adding new combinators, new parsing backends, and new language bindings. This is for developers who want to contribute to Hammer or customize it for specialized needs. If you just want to use Hammer, see Hammer Fundamentals.

---

Preliminaries

The Arena Allocator (HAllocator)

Most Hammer functions that allocate memory accept an HAllocator *mm__ parameter. This is an arena-style allocator that allows batch deallocation of parser-related memory. You'll see it throughout the internal APIs.

• The default allocator is system_allocator, which wraps malloc/free.
• The __m variants of public functions (e.g., h_foo__m) accept an explicit allocator; the non-__m variants use system_allocator.
• See the HACKING file in the Hammer source tree for more detail.

Style Conventions

• Use stdint.h types (uint8_t, int64_t, etc.) instead of bare int or long.
• Follow the existing code style in src/ for consistency.

---

Adding New Combinators

Combinators are declared in src/hammer.h, defined in src/parsers/, and listed under the parsers key in src/SConscript.

1. Declare the Combinator

Use the HAMMER_FN_DECL family of macros in hammer.h. What you're declaring is a function that returns an HParser *.

2. Implement the Constructor

The public function calls an __m variant that calls h_new_parser to instantiate the HParser * with the combinator's vtable:

HParser* h_foo(const int n) {
    return h_foo__m(&system_allocator, n);
}

HParser* h_foo__m(HAllocator* mm__, const int n) {
    return h_new_parser(mm__, &foo_vt, (void*)(intptr_t)n);
}

h_new_parser expects its third argument to be a void *. If the combinator has more than one argument, define an "environment" struct to hold them. See src/parsers/token.c for a simple example.

3. Define the Vtable

Each combinator's vtable is a struct with the following function pointer members:

.parse

static HParseResult* parse_foo(void *env, HParseState *state)

• Destructures the environment (if needed).
• Primitive combinators (e.g., h_ch): consume input directly using h_read_bits.
• Higher-order combinators (e.g., h_sequence): apply component parsers using h_do_parse.
• Constructs an HParsedToken * result and wraps it in an HParseResult *.

.isValidRegular

• Primitive combinators: use h_true.
• Higher-order combinators that can never be regular (e.g., h_indirect): use h_false.
• Higher-order combinators that might be regular: implement static bool foo_isValidRegular(void *env).

.isValidCF

• Primitive combinators: use h_true.
• Higher-order combinators that can never be context-free (e.g., h_length_value): use h_false.
• Higher-order combinators that might be context-free: implement static bool foo_isValidCF(void *env).

.desugar

static void desugar_foo(HAllocator *mm__, HCFStack *stk, void *env)

Converts the parser to a sum-of-products representation for context-free backends. Helper macros are defined in src/backends/contextfree.h. Not needed if .isValidCF = h_false.

.compile_to_rvm

static void foo_ctrvm(HRVMProg *prog, void *env)

Converts the parser to instructions for the regex VM. Instruction definitions are in src/backends/regex.h.

---

Adding New Parsing Backends

1. Register the Backend

• Declare a new extern HParserBackendVTable in internal.h alongside the existing ones.
• Add it to the HParserBackend enum in hammer.h.
• Add it to the *backends array in hammer.c.
• Add the source file to the backends list in src/SConscript.

2. Implement the Vtable

The backend vtable is a struct with three function pointer members:

.compile

int h_bar_compile(HAllocator* mm__, HParser* parser, const void* params)

Performs any setup work for the backend (e.g., generating parse tables). Returns 0 on success, -1 on failure.

.parse

HParseResult *h_bar_parse(HAllocator* mm__, const HParser* parser, HInputStream* stream)

Runs the compiled parser against an input stream.

.free

void h_bar_free(HParser *parser)

Cleans up anything that .compile created.

---

Adding New Language Bindings

For a complete walkthrough — including SWIG typemaps, SConscript templates, test reporter integration, and a contributor checklist — see the dedicated guide:

Adding a New Binding

---

Back to: Hammer Fundamentals · Home