v1.22.0

This release fixes six bugs found in a review of the parser's structure and simplifies the code around them. Most of the fixes change what some grammar matches or loads, so please read the first section before upgrading.

⚠️ Changes to be aware of

Numeric escapes in grammar text

An escape in a literal or a class takes only as many digits as the Char rule of the PEG grammar allows, and a digit after it is a character of its own. The grammar text was read by that rule, but the escape was resolved by reading every digit after it.

• \x takes one or two hex digits: '\x412' is A2, where it used to be U+0412.
• An octal escape takes up to three digits, and three only when the first is 0 to 3: '\1014' is A4 and '\477' is '7. 8 and 9 are not octal digits: '\78' is the bell character followed by 8, where it used to be @.
• \u takes six digits when they start with 0 or 10, and four or five otherwise: '\u00004142' is A42, and 'ჿFF0' is U+10FFFF followed by 0. A \u escape past U+10FFFF used to make the literal empty, so 'ሴ56' matched the empty string. It is now U+12345 followed by 6.

A grammar blob holds literals and classes with their escapes resolved, so a blob written by v1.21.0 from such a grammar keeps the old meaning. Write it again with serialize_grammar().

Cut before a label

'x'^l is a choice of its own ('x' / %recover(l)), and it cleared a cut made before it. With S <- 'a' ↑ 'x'^l 'y' / 'a' 'x' 'z', axz was accepted by the second alternative. It is now rejected, as the cut says.

no_whitespace on a macro

{ no_whitespace } on a macro was ignored unless the macro was left-recursive. With P(X) <- X X { no_whitespace } and a %whitespace rule, P('a') matched a a. It now matches aa only.

Left recursion after a prefix that can match empty

A rule is left-recursive when what comes before the recursive call can match empty. The check missed two such prefixes: a repetition that must repeat something that can match empty, such as ('x'?){2}, and a choice whose last alternative cannot match empty while another can, such as ('x'? / 'w'). So A <- ('x'?){2} A 'y' / 'z' and A <- ('x'? / 'w') A 'y' / 'z' rejected zy and accepted xzy. Both now match what A <- 'x'? A 'y' / 'z' matches: zy is accepted and xzy is rejected. A left-recursive rule grows as far as it can, so the inner A takes zy and leaves no y for the outer one.

Grammar blobs

load_blob() and GrammarBlob::deserialize() check the whole blob. A string that runs past the end, operators nested more than 2048 levels deep, bytes after the last rule, an undefined start rule and a reference to an undefined rule are errors. A blob with its last byte cut off used to load. When load_blob() returns false, the parser keeps the grammar and the start rule it had.

Blobs written by v1.21.0 still load: the format did not change.

API and ABI

• Definition can be moved but no longer copied. A copy shared the rule with the original and left the original pointing at the copy, which dangled once the copy was destroyed. peg::parser is still copyable.
• Action and Predicate can be copied, moved and assigned in a chain (f = g = ...). Copying one used to fail to compile.
• New: peg::kFail, the length a failed match returns, for user-defined rules (usr) and leave handlers. peg::success(len) tests for it.
• parse_hex_number and parse_octal_number take the maximum number of digits to read.
• The vtables of the analysis visitors (DetectLeftRecursion, TokenChecker and others) changed, and functions of Context and ParserGenerator were added or removed, which changes the ABI.

🐛 Fixes

• resolve_escape_sequence keeps a trailing backslash instead of reading past the end of the string.
• The load error for a range such as [b-a] is spelled character range is out of order....

🔧 Internals

• The operator types are listed once, and the Ope::Visitor defaults, the tracing names and accept() are generated from that list.
• The blob writer is a visitor that does not compile when an operator type has no case, where a missing case would have dropped the operator from blobs silently. The blobs of the repository's grammars are byte-identical to those of v1.21.0.
• A test checks that the two analyses that decide whether a rule can match empty agree.