v1.26.0

This release parses faster in every configuration measured, with or without packrat parsing and with or without an AST. It has source-incompatible changes to the rule settings enter, leave, error_message and the flags, and an ABI change, so it is a minor release.

⚠️ Changes to be aware of

Rule settings are tracked types

The rule settings are now types that report every change to the parser, so a parse that follows a change sees it. The types are RuleCallback (enter, leave), RuleMessage (error_message) and RuleFlag (disable_action, no_whitespace, and the other flags). Most code compiles as before:

• Assigning a lambda, a function pointer, nullptr or {} to enter and leave, calling them, testing them with if, and comparing them with nullptr.
• Assigning a const char *, a std::string, or += with a string to error_message, and passing it as const std::string & or by value.
• empty(), size(), length(), clear(), c_str(), ==, !=, <<, and converting to std::string_view.
• Assigning true or false to a flag, and reading it as bool.

These no longer compile, because a change made through a reference cannot be seen by the parser:

• Taking a reference to the stored value, such as std::string &m = r.error_message; or bool &b = r.disable_action;. Take a copy, or a const reference, instead.
• error_message.append(...), error_message += 'c', and assigning a std::string_view to error_message. Use error_message = std::string(...) or += with a string.
• flag |= true and similar compound operators on a flag. Write flag = flag || value.

ABI

The rule settings change the layout of Definition, so code compiled against an earlier version must be rebuilt.

Memory kept by a thread

Each thread keeps what the next parse on it can reuse: up to 256 value frames, and up to 64 MB of AST nodes per node type. The memory is freed when the thread exits. Parses on a thread that is short-lived or memory-limited may see this as a fixed cost.

⚡ Performance

Time of one parse, with the time to free its tree where the tree is built, against v1.25.0. Built with -O3 -DNDEBUG and measured on the same day, interleaving the two versions. The times are the minimum of three runs, except the sql.peg rows, which are the minimum of two. The DuckDB grammar is DuckDB's PEG grammar ported to cpp-peglib.

Workload	v1.25.0	v1.26.0
DuckDB grammar, big.sql (726 statements), packrat, enable_ast(true)	360 ms	184 ms (-49%)
same, time to free the trees	9.5 ms	5.2 ms
DuckDB grammar, big.sql, no packrat, no AST	382 ms	217 ms (-43%)
DuckDB grammar, TPC-H 22 queries, packrat, enable_ast(true)	4.81 ms	2.43 ms (-49%)
DuckDB grammar, TPC-H 22 queries, no packrat, no AST	4.62 ms	2.71 ms (-41%)
DuckDB grammar, one TPC-H query (Q1), packrat, enable_ast(true)	0.216 ms	0.109 ms (-50%)
benchmark/data/sql.peg, big.sql (one statement), packrat, enable_ast(true)	56.0 ms	40.9 ms (-27%)
same, time to free the tree	5.2 ms	2.0 ms
same, no packrat, no AST	34.4 ms	25.5 ms (-26%)

• Peak memory of one parse is about the same in each row.
• The instruction counts fall by about the same ratios as the times.

🔧 Internals

• Parses reuse the value frames of the previous parse on the same thread, and AST nodes are allocated from a per-thread pool.
• Rules are not scanned on every parse. The parser scans them again only after a rule setting changes.
• Matching is split into one function per kind of rule, and chains of rules that only forward a match are entered in one step.
• New tests cover the rule settings as types, the reuse of value frames after an exception or a nested parse, and trees made on one thread and released on another.