lexindex v4.5.3

Added

• CPython 3.15 and 3.15t. One cp315-abi3.abi3t wheel (PEP 803) for free-threaded 3.15 and
  later, which a GIL 3.15 also takes; the cp311-abi3 wheel still serves 3.11–3.14 and cp314t
  stays. CI runs the Python tests on 3.15, and on 3.15t with the GIL off. PyO3 0.29 → 0.30.

lexindex v4.5.2

Fixed

• A crafted DictIndex blob whose restart keys run out of order could make a routed lookup
  panic. Since 4.4.0 route_microblocks sends a lookup to its microblock by words it reads off
  each block's restart keys, and the route counts the bytes the probe shares with the word below
  it. That count stays inside the probe only while the words ascend, and nothing checked that they
  did: in a blob libFuzzer's daily parse_dict run found, the word taken as below was above and
  agreed with the probe's zero padding, so the scan began past the probe's end. A debug build
  panicked on the subtraction; a release build compared the stored suffix against more padding,
  and a dictionary in a packed code slices the probe from there, which panics in any build. Every
  read is bounds-checked, so none left the probe. Words out of order now leave their block to its
  restart run. Only a dictionary route_microblocks was called on was exposed, a
  HashedDictIndex's included; an index this crate built always has its words in order, so none
  changes. The blob is kept with a test that fails without the check
  (tests/data/panicking-4.5.0-dict-route.bdx).

lexindex v4.5.1

Fixed

• A DictIndex lookup at ten million keys and block 32 is back at 4.4.0's time; 4.4.1 had made
  it up to 23 % slower. 4.4.1 gave every run of two or more blocks whose samples tie a trie of
  their words — a threshold measured on file paths, whose runs are 183 and 3 572 blocks long. Where
  runs are short, the trie's reads (its roots,
  a node, three edge arrays megabytes apart) replaced two or three head compares on adjacent lines:
  over ten million DNA reads at block 32 all 312 500 blocks sit in 65 536 runs of 3–7, so every
  lookup took a trie, 6.3 MB of them, for 6.4 more L2-missing fills and 2.3 more page walks a probe.
  Only a run of eight blocks or more gets a trie now. In a quiet A/B, three processes a variant,
  alternated (bench/results/ties-ab-2026-09-25-arz-2be75f3.txt): DNA 724 → 595 ns (4.4.0: 588),
  the routed layout 705 → 605 (600), English titles 677 → 661 (646), URLs 774 → 747 (738); paths at
  block 256 keep 4.4.1's gain, 700 against 4.4.0's 782; and DNA at 1024, where no samples tie, reads
  472–475 in all three. A minimum of 32 read as 8 does on titles and URLs, 660 and 749. Blobs are
  unchanged.
• On a 32-bit target, a crafted blob of a hash index could make a load panic rather than fail.
  The count of the perfect hash's remap low words was multiplied in usize, which wraps there for a
  header claiming enough entries over a wide enough image; the table came out short, and only the
  bounds checks on its reads kept a load inside it, by panicking. The count is now checked, and such
  a blob is refused as too large — which the remap's reads, unchecked since this release (below),
  depend on. A property test holds the count to the exact one worked out in 128 bits.

Changed

• DictIndex builds 9–33 % faster, on every one of 13 corpora at blocks 32, 256 and 1024, blobs
  byte for byte the same. A build already ran on every thread; the gain is in the part no thread
  shared. Training draws
  its samples from the runs it keeps rather than walking every key; the order check and the key
  view's page faults run on every thread; the phrase trie a round builds is laid out 64 slots at a
  time and reads a candidate's head in one load; and a round ranks its candidates by merging the
  partitions' sorted leads. In a quiet A/B, one build a process, 11 rounds alternated
  (bench/results/dict-build-ab-2026-09-25-arz-8c99766.txt): English titles at block 32 397 →
  349 ms, Chinese titles 272 → 224, URLs 404 → 367, numbers 85 → 60 — the least gain 9.3 % (URLs),
  the most 32.5 % (numbers at 1024). ART, the quickest builder in the string-index suite, built the
  English titles in 161 ms in its 4.5.0 campaign, where lexindex's quickest took 400: the gap is
  narrower, not closed.
• DoubleArrayIndex builds in under half the time, and maps its file as fast as darts-clone opens
  one. Over jieba 0.42.1's 349 045 words
  (bench/results/cjk-build-load-2026-09-25-arz-6623184.txt), a build takes 0.159 s where 4.5.0
  took 0.354, 6.49 G instructions → 2.49 G: the trie is built from the keys in byte order, each
  decoded once; a narrow row's search tests a sparse stretch of the array word by word; and the
  placement tests four words an instruction where the CPU has AVX2. darts-clone still builds in
  0.055 s and cedarwood in 0.099 — 2.9× and 1.6× quicker. A load's walk over every slot, which is
  what lets no query check a bound, now runs on the slots' 32-bit halves, eight slots an AVX2
  instruction or four with SSE2, 15.0 M → 2.7 M instructions: load_mmap 1.03 → 0.62 ms in a
  pairwise run against darts-clone's open at 0.59–0.66 in the same processes, and 0.70 against
  0.81 in the table's run; load 2.1 → 1.6. The same slots are refused, and blobs are byte for byte
  4.5.0's.
• The perfect hash's lookup: a key the first level bumps runs 22.5 % fewer instructions, and the
  single lookup is ahead of PtrHash's fast set at a million keys where it was level. The seed's
  shift is added to the hash before its field is cut out, one instruction fewer on every placed key.
  A bumped key decodes its later level through the step table the first level ships; counts the
  remap's window with popcnt where the CPU has it, found by std's cached detection — every other
  target and Miri run the portable build, and a property test holds the two to the same answers;
  and reads the remap's tables without a bounds check past the one on its entry, on lengths and
  samples the loader has validated. Per key at a million keys, the harness's loop included: 284.8 →
  220.7 instructions a bumped key, 24.7 → 22.8 over all keys, and 2.85 fewer in the id of
  ClosedHashIndex, CompactHashIndex, HashedDictIndex and PerfectHashIndex. In a cool A/B of
  bench/mphf_vs, one lookup thread, six processes a commit
  (bench/results/mphf-cool-ab-2026-09-25-arz-e523ecd.txt): the single lookup 1.8 → 1.65 ns at
  1 M against fast's 1.8, and 2.5 → 2.4 at 10 M against 2.9; index_all 2.0 → 1.9 and 2.5 → 2.45,
  against 2.4 and 2.8.

Documentation

• SECURITY.md counts 21 unsafe fns and 62 blocks, the three builds for an instruction the
  x86-64 baseline lacks among them — the remap's popcnt lookup and the double array's AVX2
  placement and slot check — each entered only once is_x86_feature_detected! has found it.
• The double array's build and load numbers are 4.5.1's in the README, the design notes and
  the benchmarks, whose table now says where it still loses (the build) and where it no longer does
  (the load); the perfect hash's benchmarks record the single lookup's lead.
• The frontier harness times DoubleArrayIndex, and reports a corpus past its limits as
  refused rather than as a failure; the tables list only the structures a campaign ran.

lexindex v4.5.0

Added

• DoubleArrayIndex: a character-wise double-array trie for matching a lexicon against running
  text, and the fastest walk measured over Chinese. Over UD Chinese GSDSimp against jieba 0.42.1's
  349 045-word lexicon it finds every word at every position in 10.6 ns a character, against
  daachorse's 15.5 for the same block scan, and walks a position at a time in 13.4 against crawdad's
  15.8, darts-clone's 27.8, cedar's 29.0 and StringIndex's 58.6
  (bench/results/cjk-walk-2026-09-25-arz-902e4b4.txt). A label is a character — its frequency rank
  in the lexicon, read from a table indexed by code point — rather than a byte, so a Chinese word of
  three characters is three steps, not nine, and each eight-byte slot holds everything a step reads:
  the label, whether a key ends there and whether anything continues, the child row's base and the
  key's id. A step is one load, and a match costs none past it. It answers common_prefix,
  longest_prefix, for_each_common_prefix, for_each_occurrence — every key at every position,
  the text decoded once — and id, with the ids StringIndex gives the same keys; no key(id), no
  prefix enumeration. 15.62 bytes a word on that lexicon, against 17.75 for darts-clone and yada,
  18.40 for crawdad, 47.95 for daachorse and 60.57 held in memory by cedarwood. It loses on the
  build and the load (bench/results/cjk-build-load-2026-09-25-arz-902e4b4.txt): 0.35 s over the
  lexicon against darts-clone's 0.06 and cedar's 0.10, and load_mmap, which walks every slot, 1.07
  ms against darts-clone's open, which checks nothing, 0.78. Every row goes to the lowest base
  where it fits, and floors — the lowest slot a set of labels could take never moves down — keep
  that exact fit fast on a small alphabet: 0.5 s over 480 k English words, where the plain search
  took 16.6. Limits, refused at build: 2²³ keys, 65 535 distinct characters, 2²³ slots. The blob is
  BDA1, checked slot by slot on every load, load_mmap included, so a crafted blob answers wrong
  ids and never reads out of bounds; a fuzz target (parse_double_array) and a golden blob pinned
  byte for byte. Python: lexindex.DoubleArrayIndex, whose occurrences counts characters; plugged
  into jieba it cuts in 30 % less time than jieba's own dictionary, 24 % with the HMM, and 4–6 %
  less than StringIndex.occurrences (bench/results/cjk-prefix-py-2026-09-25-arz-902e4b4.txt).
  inspect reports it; plan, Overlay and the C ABI do not take it.

Documentation

• paths is re-measured at 4.4.2 in the million-key frontier, and the Python jieba lane on
  4.4.2's wheel. paths alone, every structure, three rounds
  (bench/results/frontier-named-2026-09-24-arz-b101076.json): with 4.4.1's tie trie DictIndex
  reads 712 ns at block 256 and 758 at 1024, where the 4.4.0 campaign read 794 and 826, against
  XCDAT 15's 618 — still behind, by 1.15× at block 256 and 1.11× at block 32, where it was 1.25×
  and 1.20×. Every competitor reads 0.971–1.012 of the campaign and every size is byte-identical.
  Through Python (bench/results/cjk-prefix-py-2026-09-24-arz-b101076.txt), jieba over
  StringIndex.occurrences builds the DAG in 560 ns a character against 649 over its own
  dictionary and cuts in 27 % less time, 21 % with the HMM.
• The common-prefix table on English words compares the same work, and DoubleArrayIndex
  answers both queries fastest there. Re-measured at 4.4.2 first
  (bench/results/common-prefix-2026-09-24-arz-b101076.txt: StringIndex 561 and 285 ns against
  datrie's 544 and 279, where 3.0.0 read 636 and 357), the table timed datrie's keys alone
  against lexindex's (key, id) pairs. It now times datrie's prefix_items and
  longest_prefix_item and marisa's ids, keeps datrie's keys-only row as such, and times each
  call after an untimed pass of its own (bench/results/common-prefix-2026-09-25-arz-902e4b4.txt):
  DoubleArrayIndex 473 and 234, StringIndex 530 and 295, datrie 780 and 417, its keys alone
  344 and 237. Timed after another structure's pass instead, as before, DoubleArrayIndex's
  common_prefix reads 691 against StringIndex's 578: its array is 11.7 MB on these words.
• The Chinese walk table is re-measured with every double array, each row timed warm. The table
  it replaces ran the rows alternated, each after whatever the rows before it had left in cache,
  which favoured the rows nearest the turn of a round: it had darts-clone at 41.1 and 34.8 ns and
  cedar at 49.5 and 46.2; timed after two passes of its own they read 27.8 and 34.5, 29.0 and 36.2.
  yada, crawdad's Trie and MpTrie and daachorse join them, with a build and load table.
• The README and the docs' front page say what StringIndex is for and what it is not. It is
  the automaton index — autocomplete, fuzzy search, ordered browse — and for an exact string ↔ id
  alone HashedDictIndex is faster and DictIndex smaller, and for matching a lexicon against text
  DoubleArrayIndex.

lexindex v4.4.2

Changed

• A StringIndex prefix walk — common_prefix, longest_prefix, occurrences — takes its first
  character from a table: over Chinese text it runs 18–21 % faster. A walk reads the query down
  the transducer a byte at a time, each node's header read afresh, and a query's first character
  crosses the widest nodes, the root and the ones that spell the rest of its bytes. Over UD Chinese
  GSDSimp against jieba 0.42.1's lexicon that character took 649 of a walk's 1 025 instructions. The first walk now derives the state past every first character in the
  Basic Multilingual Plane — a bitmap of 64-code-point pages up to the highest such character, a
  count a page, a node address and output sum a character that begins a key — and a walk starts
  from there, or stops at once when no key begins with its first character; a character outside
  the plane is walked as before. 581 instructions a character. Against 4.4.1 in one binary over
  five processes (bench/results/cjk-prefix-ab-2026-09-24-arz-dda19a7.txt): 80.8 → 63.9 ns a
  character in text order and 93.5 → 76.5 shuffled, 0.79–0.80 and 0.81–0.82 of 4.4.1's time in
  every process. cedarwood 0.5.0 walks the same text in 49.5 and 46.2 ns and darts-clone 0.32 in
  41.1 and 34.8, so lexindex still loses this walk, by 1.3–1.7× and 1.6–2.2× where it lost by
  1.6–2.0× and 2.0–2.7×. On five more corpora a walk runs 22 % fewer instructions over a million
  Chinese titles, 10 % over words and Russian titles, 8 % over English titles and 3 % over URLs
  (bench/results/prefix-walk-counts-2026-09-24-arz-dda19a7.txt, counts only). The table is held
  in memory only and no blob changes: 0.29 bytes a key on jieba's lexicon, 0.085 on the Chinese
  titles, 0.03 on the English and Russian ones. The first walk derives it, not the load, so an
  index that only answers id never holds it; on the lexicon that first walk runs 2.9 M
  instructions more.

Documentation

• The common-prefix walk over Chinese text is measured against cedar and darts-clone, and
  lexindex still loses it. docs/benchmarks.md gains a section on jieba's lexicon over 4 997
  sentences: from Rust, 4.4.2's StringIndex walks 1.3–1.7× slower than cedarwood 0.5.0 and
  1.6–2.2× slower than darts-clone 0.32, holding a seventh of cedar's bytes and half darts-clone's;
  from Python, jieba over 4.4.0's StringIndex.occurrences cuts in 25 % less time than over its own
  dictionary, 18 % with the HMM, the fastest backend measured
  (bench/results/cjk-prefix-2026-09-24-arz-3a73ed5.txt).

lexindex v4.4.1

Changed

• A DictIndex lookup that lands among blocks whose samples tie is placed by a trie of their
  words: on a million file paths id takes 8 % less time, at the default block and at 1024. A
  lookup opens with a binary search over one eight-byte sample a block, taken past the bytes every
  head shares; where a run of blocks shares the probe's sample, their heads were compared one at a
  time. On paths that run is the index — 3 899 of 3 907 samples tie at the default block, paths
  sharing long prefixes — and a lookup made 11.4 head compares, a quarter of its instructions. A
  load now splits each such run by its heads' eight-byte words past what they share, and a sub-run
  that ties again the same way past its own prefix: on paths 1 136 nodes and 5 035 edges, nine
  levels at the deepest. A lookup descends it reading only the probe, checks once against the head
  it arrives at, and starts the block's scan from that compare: 16.3 % fewer instructions a member
  id at 256 and 10.8 % at 1024, and within 0.6 % — all of it fewer — on words, English titles,
  URLs and UUIDs. Every answer is the one 4.4.0 gives, and no blob changes, byte for byte. The trie
  is held in memory only: 0.10 bytes a key on paths at 256, 0.02 at 1024, under 0.005 on titles and
  URLs, nothing where no samples tie. A load of paths runs 11 % more instructions at 256 and 3 % at
  1024, and at most 1 % more elsewhere. Against 4.4.0 in one process, each build loaded twice, over
  three processes (bench/results/id-ab-2026-09-24-arz-b362f43.txt): id on paths 768 → 704 ns at
  256 and 812 → 750 at 1024, every allocation of the new build under every one of the old; on
  English titles, URLs, UUIDs and words, and every key_into, within ±1.5 %. What it costs is at
  load, where the trie is built: a mapped load of paths takes 238 µs against 130 at 256 and 118
  against 96 at 1024, and from_bytes, which reads and checks the whole blob, 4.2 and 0.2 % longer.
  Elsewhere from_bytes moves by at most 1.7 % and load_mmap by at most 5 %, 12 µs.
• A load of a DictIndex in a character code holds half the tables it did. Every load built the
  code's lookup tables beside the blob — a mapped one too — and they held the alphabet twice, took
  four bytes a codeword where sixteen bits carry it, and covered all of Unicode at their first level
  whatever the keys used. On jieba's lexicon a from_bytes index held 3.687 bytes a key, 21 % over
  its 3.040-byte blob and above marisa-trie's 3.578, whose load is its file to within 0.1 %; it
  holds 3.392 now, 5.2 % under marisa's. A million Chinese titles go from 6.474 to 6.243 bytes a key
  and a million Russian ones from 7.254 to 7.138, the tables halving on all three. Blobs are
  unchanged, byte for byte (bench/results/resident-2026-09-24-arz-b362f43.txt, which has the
  marisa rows). What it costs is in spelling the query, where a codeword is widened from sixteen
  bits: id on the lexicon runs 1 % more instructions and takes 1–2 % more time, 260 → 266 ns in
  the A/B above, and on the titles 0.6 and 0.7 %; key_into is level on all three.

Documentation

• What a mapped DictIndex builds is stated as it is. The README, the docs' front page and the
  Python module said a load_mmap builds only the per-block samples; it also decodes the symbol
  tables and, where they apply, builds a code's tables and the trie above — 0.07–0.39 bytes a key in
  all over the corpora measured, of which docs/design.md gives the parts.

lexindex v4.4.0

Added

• DictIndex::route_microblocks, in Rust and in Python: a faster DictIndex search for eight
  bytes of memory a microblock. Once the samples and the head have chosen a block, a lookup walks
  the block's restart run — the first keys of its microblocks, coded against one another — to find
  the microblock it scans. route_microblocks() derives, for every block, the bytes its restarts
  share with its head and each restart's eight bytes past them; a lookup then counts the words below
  its own instead of walking, and walks only where a word ties it. No answer changes, and no blob,
  byte for byte: the words are held in memory only, 8 / micro bytes a key — 0.5 at the default
  block of 256, 0.25 at 1024 — and the method returns the bytes they take. It is a method rather
  than the default because it is paid for twice. Routed, the default block would give up its size
  lead over the smallest trie on four corpora of the thirteen at a million keys — words,
  domains, idents and pypi, where it leads by 0.34–0.39 B/key — and deriving the words reads
  every block's restart run, which a lazy load_mmap leaves unread until a lookup needs it. It
  takes &self, so a dictionary shared through an Arc — the Python object,
  HashedDictIndex::dict — routes in place, and a lookup that races it walks the run. Against the
  same index unrouted, in one process and in two builds of the harness
  (bench/results/id-ab-2026-09-24-arz-0216b39.txt): at the default block id takes 19–31 % less
  time on twelve corpora at a million keys and 13–22 % on six at ten million, and ids_of 16–24 %
  and 7–17 %; at 1024 keys a block id takes 23–36 % and 14–28 % less. With the change below,
  against 4.3.3, a routed id at the default block takes 23–41 % less time at a million keys and
  19–31 % at ten million. paths, whose restarts tie in the eight bytes past what they share, moves
  between −2.4 and +1.8 %. At 32 keys a block, two microblocks, the words save at most 15 %, and
  cost numeric at ten million 3–10 %.

Changed

• A DictIndex lookup is faster with nothing asked for: id by 4–16 % and ids_of by 4–13 % on
  twelve corpora of thirteen at a million keys. Most of a lookup's instructions go to two
  front-coded scans, and the scan matched on the shard's codec for every entry it compared. It now
  picks the codec and the kind of header code once a run and walks the run in a loop of its own for
  that pair; reads the probe's last eight bytes once a walk, where every compare that reached the
  end of the key had folded them a byte at a time; and tests the symbol table's own codes before its
  escape and phrase prefixes. Blobs are unchanged, byte for byte. Against 4.3.3 in the run above,
  unrouted: id takes 4.9–15.5 % less time at the default block on twelve corpora at a million keys
  and 2.3–11.7 % on six at ten million, 6.3–16.4 % and 2.8–15.7 % at 1024 keys a block, and
  3.7–15.2 % and 0.6–14.0 % at 32; ids_of at the default block 3.7–12.7 % and 2.2–9.6 %. On
  paths every lookup is 1.1–2.6 % faster.

lexindex v4.3.3

Fixed

• DictIndex::id on a blob in a character code is as fast as 4.2.0's again. 4.3.2's id was
  6 % slower than 4.2.0's on a million Chinese titles. Counted per query, the search was not the
  cause — in code it costs fewer cycles than 4.2's search in UTF-8 — but spelling the query: 342
  instructions and some 130 cycles, run before the first step of the search could issue. The code
  keeps order, so the blocks a query falls between are the same whether the per-block samples are
  taken from the heads as coded or as decoded, and a coded blob's loader now takes both. Where the
  decoded samples tie no more blocks than the coded ones — 51 of 3 907 against 70 on the Chinese
  titles, none either way on jieba's lexicon — id searches them with the query's own UTF-8 and
  spells it while that search waits on its loads. The Russian titles tie 659 decoded against 161
  coded, eight bytes of UTF-8 being four letters, and keep the coded route. The decoded samples are
  derived at load like the coded ones and take as much memory, eight bytes a block; blobs are
  unchanged, byte for byte. With the change below, against 4.2.0 and 4.3.2 in one process, in two
  builds of the harness (bench/results/id-ab-2026-09-24-arz-af19cb7.txt): on the Chinese titles
  id is level with 4.2.0 (−0.5 and +0.3 %) and 5.7–6.5 % faster than 4.3.2, on the lexicon
  6.7–8.0 % faster than 4.2.0, and on the Russian titles 2.7–3.0 % faster than 4.3.2 and 0.8–2.7 %
  slower than 4.2.0.
• DictIndex::ids_of on a blob in a character code is 6–13 % faster. It checked every &str
  key as UTF-8 a second time, and grew, zero-filled and cut back an arena the size of the whole
  batch a key at a time. It now takes a str as it is, and spells 32 keys at a time into one arena
  reused across the batch, so it no longer holds a spelled copy of every key. Against 4.3.2 in the
  same run: 8.1–8.3 % faster on the Chinese titles, 12.6–12.9 % on the Russian ones and 6.1–6.7 % on
  the lexicon. That still leaves it 2.8–4.5 % and 1.2–2.4 % slower than 4.2.0's on the two sets of
  titles, and 5.0–5.3 % faster on the lexicon: on all three corpora a batch is slower than a loop of
  id in every version measured, so the spelling has no wait to run beside.

Changed

• A DictIndex lookup compares a head only where the samples leave it open, and id is faster
  on every corpus but paths. A block whose sample is below the probe's has its head below the
  probe, and one whose sample is above has its head above, so only a run of blocks sharing the
  probe's sample leaves heads to compare. Every lookup had also compared the head of the block
  before that run: one more dependent load and a memcmp, for a block the samples had already
  chosen. Between 0.4 and 12.4 % of the probes land in a run on thirteen of the fourteen corpora
  measured; on paths all of them do, since every path shares one byte and past it the samples tie
  3 899 blocks of 3 907. On the eleven corpora that take no code, against 4.3.2 in the same run,
  id is 2.7–5.5 % faster on eight of them, 0.9–4.3 % on identifiers and UUIDs and level on paths;
  ids_of is 2.8–5.2 % faster on nine and level on paths, and on identifiers 4.0 % faster in one
  harness build and 1.5 % slower in the other, whose whole row, the control's included, the machine
  slowed by 9–25 %.

lexindex v4.3.2

Fixed

• DictIndex builds faster than 4.2.0 again; 4.3.0 had made every in-memory build 11–81 %
  slower. Before building, 4.3.0 counted the keys' bytes outside ASCII to learn whether a
  character code could pay, and where one could it counted their characters and spelled every key:
  three passes on one thread, in key order, over the caller's keys. Those lie wherever the caller
  allocated them, so such a pass waits on memory at every key, and the byte count compiled to four
  instructions a byte, which keeps few of those loads in flight. The count now takes eight bytes a
  step; the three passes run a range of keys a thread, over one view of the keys that the
  encoder's threads now share rather than taking their own; and every pass prefetches the key 32
  places ahead. The order check and the deduplication in front of the build — a sort's check for
  a run and a dedup_by, two more passes of the same kind — are now one pass that reads ahead.
  Blobs are unchanged, byte for byte. Over the thirteen corpora of the million-key frontier, at
  blocks 1024 and 256, alternated with 4.2.0 and 4.3.1 in one run
  (bench/results/build-ab-2026-09-23-arz-152a00f.txt), a build takes 11–53 % less time than
  4.3.1's on every corpus; against 4.2.0's it is 29–35 % faster on Russian titles, 11–14 % on
  Chinese ones, 3–25 % on eight Latin corpora, and within 2 % on paths, English titles and URLs.
  The loss was found by re-measuring the frontier at 4.3.1
  (bench/results/frontier-1m-2026-09-23-arz-8dc8b73.json), where DictIndex no longer built
  first on three corpora of the thirteen, and the corpus sweep at 4.3.1 shows it too
  (bench/results/sweep-2026-09-23-arz-8dc8b73.json).

lexindex v4.3.1

Fixed

• key_into on a blob in a character code is back at 4.2.0's speed, and id much closer to
  it. 4.3.0 decoded a key a character at a time through the tables the code is stored in — the
  codeword's lead found among the leads, a branch on the mode and on the lead's width,
  char::encode_utf8 — and then checked the finished key as UTF-8 a second time; it spelled a
  query through a two-level index, behind a branch on the mode. A codeword now decodes in two reads
  and one four-byte store, out of a table of the 256 values its first byte can take and one of
  every character's UTF-8. What the decoder writes is whole characters out of a table built from
  chars, so it becomes a String without the second pass; a debug build still checks it, and so
  do the fuzz targets. Code points below U+0800 — ASCII, Cyrillic and every other script UTF-8
  spells in two bytes — are one flat level of the index, and under the eight-bit mode ASCII is in
  the index too, so spelling a query no longer asks which mode it is in. Blobs are unchanged, byte
  for byte. Against 4.2.0 in one process, in two builds of the harness
  (bench/results/charcode-ab-2026-09-23-arz-668a1d3.txt): key_into is 2.5–3.3 % faster on a
  million Russian titles, where 4.3.0 was 19–23 % slower, within 2 % on a million Chinese ones
  (4.3.0: 11–12 % slower) and level on jieba's lexicon (4.3.0: 4–5 % slower); id is 1–2 %
  slower on the Russian titles (4.3.0: 6–7 %), 6–8 % on the Chinese ones (4.3.0: 12–13 %) and 3–4 %
  faster on the lexicon, as in 4.3.0. English titles, whose blob is 4.2's, read within 2 % of
  4.2.0, the band two builds of the harness move by. Most of what id still pays is spelling the
  query: all of it on the Russian titles, where spelling it twice adds 610 instructions and id
  pays 605 over 4.2.0, and two thirds of the 530 instructions it pays on the Chinese ones.

Documentation

• The million-key size table and the README's margin over marisa-trie are re-measured at
  4.3.0 (bench/results/sweep-2026-09-23-arz-4f2a0e9.json), so they show the character code. Of
  the 264 cells the run shares with the 4.0.0 one, 252 are the same to the byte; the twelve that
  moved are DictIndex on the Russian and Chinese titles, 6.7–14.7 % smaller, and their margins
  over marisa's best setting go from 2.2 and 1.6 % to 14 and 9 %, so the README's range now starts
  at English titles' 3.9 %. At ten million keys
  (bench/results/sweep10m-2026-09-23-arz-4f2a0e9.json) all 66 cells are the same to the byte: the
  code takes none of those six corpora. The run's timings are not published, since the machine was
  busy.