Engine performance release: measured throughput gains across the hot grammars (real-Swift +6~9% cumulative, JavaScript/TypeScript similar), three fidelity/safety fixes for custom grammars, and a grammar-compilation leak fix. Verified against highlight.js 11.11.1: 375-fixture corpus token-exact, 5-grammar randomized differential at 0 divergence, ThreadSanitizer-clean, 260-test suite green.
- Fixed: a keyword listed under two scope groups of the same mode now
resolves to a deterministic winner (sorted scope order) instead of
varying with SwiftDictionaryiteration order across processes.
highlight.js resolves such duplicates by object insertion order, which
the grammar model cannot observe — see FIDELITY.md. No bundled grammar
declares a duplicate. - Fixed: four matcher prefilters whose ASCII candidate scans cannot see
ICU case folding (identifier-before-colon, number literals, the Swift
uppercase lead-in, and word-before-paren) now decline case-insensitive
rules and fall back to raw ICU enumeration. All bundled grammars
declare these rules case-sensitively; this hardens the public
custom-grammar API only. - Fixed: grammar compilation leaked raw-mode reference cycles that
variant expansion had detached from the grammar root (self-recursive
variant-carrying modes, e.g. Scheme's nested lists — ~90 KB leaked per
cold 65-language auto-detection). Teardown now reaches every mode
considered for expansion explicitly; a weak-reference regression test
pins the exact shape. - Engine: bare
\s+rules synthesize ASCII whitespace runs directly and
defer to ICU whenever a non-ASCII unit could participate (\sis
Unicode-aware). Measured (15-pair paired A/B): JavaScript incremental
latency −1.37% (95% CI excluding zero, 15/15 pairs); no regressions. - Engine: keyword lookups probe a flat UTF-16 table straight from the
input buffer — no per-word substring,Stringallocation, or Unicode
hashing — and relevance-saturation counters are dense per-language
arrays instead of a secondString-keyed dictionary. Case-insensitive
words containing non-ASCII units keep the full Unicode-folding path.
Measured (15-pair paired A/B, 95% CI excluding zero): SQL +2.26%,
real-Swift +1.14%, TypeScript +1.00%, JavaScript +0.93% throughput;
controls neutral.
Behavior note for custom grammars: case-sensitive keyword matching is
now UTF-16 code-unit exact, matching highlight.js's JavaScript object
lookup. Previously, SwiftStringcanonical equivalence let an NFC
keyword match NFD input — an accidental divergence from upstream. All
bundled grammars use ASCII keywords and are unaffected. - Engine: the bare ECMAScript identifier rule is matched by an exact
raw UTF-16 scan (its character classes are pure ASCII), eliminating
whole-window ICU enumeration of every identifier. Case-insensitive
rules keep ICU (its case folding reaches non-ASCII input units).
Measured (15-pair paired A/B, 95% CI excluding zero): JavaScript
+2.83%, TypeScript +1.99% throughput, JavaScript incremental −0.73%;
non-ECMAScript controls neutral. - Engine: the literal-table prefilters (Swift punctuated keywords,
ECMAScript value starters) dispatch through a 128-slot directly-indexed
bucket array instead of hashing aDictionaryat every input position.
Measured (15-pair paired A/B, 95% CI excluding zero): real-Swift
throughput +6.13%, JavaScript +1.48%; all controls neutral. Construction
fails closed on non-ASCII first units, so rule semantics are unchanged. - Engine: cached matches carry their capture groups inline
(.none/.group1/.icu) instead of allocating an
NSTextCheckingResult(plus a range buffer) per hand-synthesized
match, and rule consultations read cached ranges without
Objective-C dispatch. Measured (15-pair paired A/B, 95% CI excluding
zero): JavaScript +4.78%, TypeScript +3.78%, real-Swift +1.82%, SQL
+0.87% throughput; JavaScript incremental latency −1.58%. No behavior
change; group semantics remain JavaScriptmatch[N]-exact.