v1.21.0

This release makes large grammars faster and fixes the bugs found on the way. Several of the fixes change what a grammar matches in corner cases, so please read the first section before upgrading.

⚠️ Changes to be aware of

Case-insensitive classes and literals

• [...]i was decided in three places that disagreed: S <- [0-Z]i accepted _ while S <- [0-Z]i+ rejected it. A class with i now matches a character when the class without i matches the character or its other case, as regular expressions do. Classes such as [a-z_]i are not affected.
• Only ASCII letters have another case, in classes, '...'i literals and dictionaries alike. What was folded outside ASCII used to depend on the locale and the platform: after setlocale() with a UTF-8 locale on macOS, [a-z]i matched the Kelvin sign and 'ét'i matched the bytes E3 A9 74. In the C locale nothing changes.
• [[:^lower:]]i and [[:^upper:]]i leave out every letter, as [^[:lower:]]i does and as Perl, PCRE2 and Go's regexp do. [[:^upper:]]i used to match everything.

Input that is not well-formed UTF-8

Bytes that are not well-formed UTF-8 (an overlong form, a surrogate, a code point past U+10FFFF, a sequence cut short, a stray byte) are no longer read as a character: ., a character class, negated or not, and a character all fail there, as in PEGTL, the Rust regex crate in Unicode mode and PCRE2 with PCRE2_MATCH_INVALID_UTF. Before, the overlong C1 81 read as A to [A-Z] but not to [A-Z]+, a lead byte with nothing after it made a negated class match zero bytes, and . stepped by the lead byte alone, so a comment rule such as (!'\n' .)* could swallow the newline after a Latin-1 byte. Errors count each such byte as one column. Grammar text that is not well-formed is a syntax error. decode_codepoint and codepoint_length follow the same rule.

precedence rules

A precedence rule failed as a whole when an operator matched but its right operand did not, so 1+ did not match E '+'?. It now stops before that operator, as the repetition ATOM (OPERATOR ATOM)* that the instruction annotates does. With an ordered choice around the rule, an input that used to fall through to a later alternative can now be taken by the precedence rule.

Keywords without %word

The pattern !KEYWORD < [a-z_]i [a-z0-9_]i* > had a fast path that compared the whole identifier against the keywords. It accepted online with KEYWORD <- 'on'i, although !KEYWORD rejects it because the keyword matches a prefix, and it applied only to grammars of a certain shape (no %whitespace, or a keyword of several words). The fast path is gone, so such a grammar now rejects identifiers that start with a keyword. Use %word, or end each keyword with a lookahead such as ![a-z0-9_]i, to match whole words.

Macros

A macro cannot be the start rule: it is a load error now, where it used to crash on parse.

Grammar blobs

load_blob() returns false for a blob written by v1.20.0 or earlier. A blob holds a class's ranges as they were expanded, and the ranges of [[:^lower:]]i written by v1.20.0 would now match every character. Write the blobs again with serialize_grammar().

ASTs

A rule renamed, or given another ast_name, after enable_ast() keeps the node name it had when enable_ast() was called.

API and ABI

• Removed: Context::tolower_table, Definition::has_macro_ref, and the argument frames of Context (args_stack, push_args, top_args and the like).
• PrioritizedChoice::opes_ is const: the alternatives of a choice are fixed when it is built.
• FindReference is now ReplaceParameters.
• New in namespace peg: to_lower(char) and other_case. Code that has functions of these names and using namespace peg may need to qualify its calls.
• The layouts of Context, Definition, Sequence and PrioritizedChoice changed, which changes the ABI.

🐛 Fixes

Macros

A macro call used to rebuild its arguments on every call and to look its parameters up through a stack of frames. A parse now makes each instance of a macro once, the first time it is called with given arguments.

• Nested call with a parameter: with M0(P) <- P and M1(P) <- M0(M0(P)), the inner argument was resolved against the wrong frame, which crashed, hung or gave a wrong result.
• Macro call as an argument: M(N('a')) and M(N('b')) shared the seeds of a left-recursive M, and the detection of left recursion looked at the first of them only, so R <- M(N('a')) 'x' / M(N(R)) 'b' / 'c' was not seen as left-recursive.
• Compound argument of a left-recursive macro: with A <- M('a' / 'b'), B <- M('a' / 'c') and M(P) <- M(P) '+' P / P, S <- A 'x' / A 'y' / B 'z' rejected a+cz.
• Back reference as an argument: S <- $c< 'a' > M($c) with M(P) <- P crashed.
• Labels: a label that failed inside an argument did not cut the choice around the macro call, and a label given as an argument (M(L) <- 'a' ^L) crashed when a logger was set.

Action of a precedence rule

vs.name() was the name of whatever rule had last used that scope, often empty, and vs.choice() could be the choice of an operand. The action now sees the rule's name and no choice, as the action of any rule whose body is not a choice.

Grammar text

S <- [\xE3\x81\\], with the two raw bytes of a sequence cut short, failed an assertion while the grammar was loaded.

⚡ Performance

Instructions of one parse against v1.20.0. The large grammar is the grammar of DuckDB's PEG parser ported to cpp-peglib (about 1,100 rules, %word, choices of hundreds of keywords), on a 1.2MB SQL file with packrat parsing.

• Large grammar, recognition: 5.05G -> 2.99G instructions (-41%), 399ms -> 295ms.
• Large grammar, enable_ast(true): 7.39G -> 4.98G (-33%), 636ms -> 491ms.
• Large grammar, enable_ast(): 9.84G -> 7.76G (-21%), 865ms -> 763ms.
• Macros: the calculator of the README's macro example on an 8.3MB expression, 12.66G -> 6.33G (-50%), 794ms -> 423ms.
• benchmark/data/sql.peg (55 rules): recognition 355M -> 332M (-6%), enable_ast(true) without packrat 1.08G -> 1.04G (-4%). This includes the cost of dropping the keyword fast path.

What changed:

• A choice reads the alternatives that can start with the next byte from a list, instead of going through all of them and skipping one at a time. Error messages are the same.
• A case-insensitive literal no longer fills a 256-entry table from the C library each time a %word check runs.
• With enable_ast(true), a node that stands in for its parent takes the parent's name and tag from the rule, without looking the name up and hashing it for every match.
• A macro call no longer rebuilds its arguments.