Skip to content

0.2.0

Latest

Choose a tag to compare

@Eversight Eversight released this 14 Jul 06:29

Engine performance release: measured throughput gains across the hot grammars (real-Swift +6~9% cumulative, JavaScript/TypeScript similar), three fidelity/safety fixes for custom grammars, and a grammar-compilation leak fix. Verified against highlight.js 11.11.1: 375-fixture corpus token-exact, 5-grammar randomized differential at 0 divergence, ThreadSanitizer-clean, 260-test suite green.

  • Fixed: a keyword listed under two scope groups of the same mode now
    resolves to a deterministic winner (sorted scope order) instead of
    varying with Swift Dictionary iteration order across processes.
    highlight.js resolves such duplicates by object insertion order, which
    the grammar model cannot observe — see FIDELITY.md. No bundled grammar
    declares a duplicate.
  • Fixed: four matcher prefilters whose ASCII candidate scans cannot see
    ICU case folding (identifier-before-colon, number literals, the Swift
    uppercase lead-in, and word-before-paren) now decline case-insensitive
    rules and fall back to raw ICU enumeration. All bundled grammars
    declare these rules case-sensitively; this hardens the public
    custom-grammar API only.
  • Fixed: grammar compilation leaked raw-mode reference cycles that
    variant expansion had detached from the grammar root (self-recursive
    variant-carrying modes, e.g. Scheme's nested lists — ~90 KB leaked per
    cold 65-language auto-detection). Teardown now reaches every mode
    considered for expansion explicitly; a weak-reference regression test
    pins the exact shape.
  • Engine: bare \s+ rules synthesize ASCII whitespace runs directly and
    defer to ICU whenever a non-ASCII unit could participate (\s is
    Unicode-aware). Measured (15-pair paired A/B): JavaScript incremental
    latency −1.37% (95% CI excluding zero, 15/15 pairs); no regressions.
  • Engine: keyword lookups probe a flat UTF-16 table straight from the
    input buffer — no per-word substring, String allocation, or Unicode
    hashing — and relevance-saturation counters are dense per-language
    arrays instead of a second String-keyed dictionary. Case-insensitive
    words containing non-ASCII units keep the full Unicode-folding path.
    Measured (15-pair paired A/B, 95% CI excluding zero): SQL +2.26%,
    real-Swift +1.14%, TypeScript +1.00%, JavaScript +0.93% throughput;
    controls neutral.
    Behavior note for custom grammars: case-sensitive keyword matching is
    now UTF-16 code-unit exact, matching highlight.js's JavaScript object
    lookup. Previously, Swift String canonical equivalence let an NFC
    keyword match NFD input — an accidental divergence from upstream. All
    bundled grammars use ASCII keywords and are unaffected.
  • Engine: the bare ECMAScript identifier rule is matched by an exact
    raw UTF-16 scan (its character classes are pure ASCII), eliminating
    whole-window ICU enumeration of every identifier. Case-insensitive
    rules keep ICU (its case folding reaches non-ASCII input units).
    Measured (15-pair paired A/B, 95% CI excluding zero): JavaScript
    +2.83%, TypeScript +1.99% throughput, JavaScript incremental −0.73%;
    non-ECMAScript controls neutral.
  • Engine: the literal-table prefilters (Swift punctuated keywords,
    ECMAScript value starters) dispatch through a 128-slot directly-indexed
    bucket array instead of hashing a Dictionary at every input position.
    Measured (15-pair paired A/B, 95% CI excluding zero): real-Swift
    throughput +6.13%, JavaScript +1.48%; all controls neutral. Construction
    fails closed on non-ASCII first units, so rule semantics are unchanged.
  • Engine: cached matches carry their capture groups inline
    (.none/.group1/.icu) instead of allocating an
    NSTextCheckingResult (plus a range buffer) per hand-synthesized
    match, and rule consultations read cached ranges without
    Objective-C dispatch. Measured (15-pair paired A/B, 95% CI excluding
    zero): JavaScript +4.78%, TypeScript +3.78%, real-Swift +1.82%, SQL
    +0.87% throughput; JavaScript incremental latency −1.58%. No behavior
    change; group semantics remain JavaScript match[N]-exact.