fix(builtins): size regex_match from the pattern; null for non-participating groups (fixes #629)#652
Conversation
… non-participating groups (InauguralSystems#629) regex_match used a fixed regmatch_t[16] and a loop whose CONDITION tested `matches[i].rm_so >= 0`. Two bugs followed: 1. POSIX sets rm_so = -1 for a capture group that did not participate (e.g. an unmatched optional `(x)?`). With that test in the loop condition, the first non-participating group TERMINATED emission, deleting every later capture: `regex_match of ["ab", "(x)?(a)(b)"]` returned `["ab"]` instead of the full match plus three groups. 2. Groups past 15 silently vanished — a 17-group pattern returned 16 elements. Both break the docs/BUILTINS.md invariant that group n sits at index n on a successful match. Fix: size the array from the compiled pattern (`re.re_nsub + 1`) via the overflow-guarded xmalloc_array (re_nsub is attacker-controlled through the pattern but POSIX-bounded), pass that count to regexec, free it on every path, and move the participation test into the loop BODY — a group with rm_so < 0 now emits `null` (distinct from `""`, mirroring Python's `(None, 'a', 'b')`) instead of stopping the loop. Existing error / no-match behavior is unchanged. Tests: the issue's two repro cases, non-participating group in the middle and at the end, and a 17-group pattern asserting all groups are present. docs/BUILTINS.md pins the null encoding and the no-group-cap guarantee. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Merging. Verified independently, including the two things this rewrite actually risks. Both bugs fixed, Python parity confirmed: Your Those two are genuinely different facts, and emitting The refcount protocol is the thing I checked hardest, because it's CLAUDE.md's first hard-won rule and The other risk is that you replaced a fixed stack array with a heap allocation. Frees on both the no-match and success paths, Your runner-tally note is right, and it's now a pattern worth fixing properly — |
Fixes #629, both bugs, with your prescribed shape: the
regmatch_tarray is sizedre.re_nsub + 1(overflow-guarded through the repo's ownxmalloc_array/safe_size_mulidiom, freed on both the no-match and success paths),regexecgets that count, and the participation test moved from the loop condition into the body — a non-participating group now emitsnullinstead of terminating emission.Repro → fixed
On main:
regex_match of ["ab", "(x)?(a)(b)"]→["ab"](matched, zero groups) and a 17-group pattern returns 16 elements. On this branch:["ab", null, "a", "b"](Python parity:(None, 'a', 'b')) and the 17-group pattern returns all 18.Why
nullfor absent groups: it's the language's own precedent —VAL_NULLis first-class,builtins.c:5514/5540already appendmake_null()into result lists (a borrowed singleton, solist_append_ownedis leak-safe by the existing pattern),assert_eqagainstnullis an established test idiom, and it stays distinguishable from""(a group that matched empty). Pinned indocs/BUILTINS.mdalong with the no-group-cap guarantee.Tests + verification
New
#629marker block intests/test_regex.eigs(15→34 checks): the issue's repro + control, non-participating group in the middle and at the end, and a 17-group pattern with groups 16/17 present. Mutation-checked twice (implementer, then independently re-run with a rebuild between every swap): reverting onlysrc/builtins.cfails the new asserts with the truncated results; restored, all pass. Full suite 2890/2890;make asan(withdetect_leaks=1) clean over the repros and both regex test files — the dynamic-allocation path specifically; no new-Wall -Wextrawarnings;--linton the touched.eigsclean (the C file isn't lintable by the .eigs linter, as established).One runner note, deliberately not touched (scope):
run_all_tests.sh:1260hard-codes "Regex (15 checks)" as that slot's declared tally — the 19 new asserts run and gate (the block is all-or-nothing on its pass marker) but the reported total stays 2890. Happy to bump the declared count in a follow-up or here if you prefer.🤖 This PR was written with AI assistance (Claude), directed and verified by a human-supervised workflow.