Add: Exact Levenshtein search for immutable dictionaries

[grouville]

This PR adds exact bounded Levenshtein search for immutable string dictionaries. It targets applications that build a dictionary once and query it many times, returning every matching original ID and exact distance within an inclusive bound.

It addresses #243. That issue points to Levenshtein automata as a faster alternative to scanning every string. The implementation started there, then kept the automaton only where it helps. A small bound can reject most dictionary prefixes early. A wide bound reaches most branches, so carrying automaton state through the whole tree becomes more expensive than scanning the dictionary directly. The public API stays the same while the index chooses the exact path for each query.

What this adds

import stringzillas as szs

index = szs.LevenshteinIndex([b"book", b"back", b"book", b"boon"], max_distance=2)
query_ids, dictionary_ids, distances = index([b"cook", b"book"], bound=1)

assert sorted(zip(query_ids.tolist(), dictionary_ids.tolist(), distances.tolist())) == [
    (0, 0, 1),
    (0, 2, 1),
    (1, 0, 0),
    (1, 2, 0),
    (1, 3, 1),
]

The index owns a copy of the dictionary. IDs always refer to the original input order, so duplicate strings remain separate results. Search returns every match within bound; result order is unspecified.

The same operation is available as:

• levenshtein_index and levenshtein_index_utf8 in C++;
• opaque index and reader handles in C;
• LevenshteinIndex and LevenshteinIndexUTF8 in Python.

The C interface follows the existing StringZillas batch model. It accepts sequence, 32-bit tape, and 64-bit tape collections, supports caller-owned result allocation and output sizing, and can search on one or several CPU cores.

How search is selected

Query	Exact path
k=0	Hash lookup followed by a complete string check
k=1..2, when deletion records stay small	Shared deletion variants followed by exact verification
Longer strings or selective larger bounds	Compressed prefix-tree traversal with bounded distance state
Wide bounds	One query mask reused while the owned dictionary is scanned with bit-parallel distance

The automatic builder first estimates the deletion records required for the whole dictionary. At k=1..2, it uses that index only when the estimate stays below 80 records per word on average. Otherwise it builds the prefix tree for the whole dictionary. C++ callers may give an explicit word-length cutoff when they want a mixed index. The value 80 is an empirical memory guard, not a claim that one plan is best for every workload.

The deletion and tree paths are lossless candidate filters. Every candidate is verified against the owned dictionary text before it is returned. Hash collisions, repeated deletion variants, and cached tree states can add work, but cannot change the answer. Construction also happens in a temporary index, so an allocation or validation failure leaves the previous index unchanged.

The deletion filter follows the lossless observation in Improved Fast Similarity Search in Dictionaries. This is not a direct implementation of that paper. It combines deletion filtering with a compressed prefix tree, cached distance steps, and a direct bit-parallel crossover path.

Text, ownership, and concurrency

The byte index treats each byte as one symbol. The UTF-8 index validates and decodes the dictionary once, validates every query, and measures edits between Unicode codepoints. It does not normalize text, fold case, apply locale rules, or count grapheme clusters.

The index is immutable after construction. C++ keeps mutable search memory in independent readers, so several readers can safely share one index. C and Python keep that memory behind the handle; concurrent callers should use separate handles. The implementation is CPU-only. GPU device scopes are rejected instead of silently running elsewhere.

Where this fits

This API is for repeated complete retrieval over a fixed dictionary. It fits spelling candidates, catalogue matching, and other services that need every dictionary entry inside a known edit radius.

Auditing existing fuzzy-search projects was useful because most did not actually have that contract. Some use normalized similarity, weighted edits, token scores, dense pairwise matrices, or only the nearest result. Replacing those calls with this index would change behavior. The audit therefore narrowed the API rather than producing a misleading production migration claim.

Existing boundaries

• The maximum bound is chosen when the index is built. A query may use that bound or a smaller one.
• The dictionary cannot be updated in place and the index has no persistent on-disk format.
• Search returns all bounded matches. It does not yet provide nearest or top-N search, including the unbounded case mentioned in the issue.
• Distance is unit-cost plain Levenshtein. Weighted distance, normalized similarity, Damerau transpositions, token similarity, and ranking belong to different contracts.
• This is not the dense all-pairs within(k) experiment. That work was removed because it solved a separate problem without a concrete caller.

Correctness and failure behavior

Final reviewed head: 19b7faed60051867cb3bf8b1367d3da84bc1b209.

The C++ suite compares 2,548,910 exhaustive dictionary memberships with an independent dynamic program. It covers duplicate IDs, embedded zero bytes, every search path, bounds through 40, and strings around the 64, 128, and 512-symbol block boundaries. Separate tests cover C, Python, UTF-8 validation, failed rebuilds, allocator failures, and independent readers.

Source review after the first draft found three real failure-path bugs:

• search allocation failure could look like a valid missing match;
• a retried Python output allocation could release the same arrays twice on an overflow branch;
• queries= was rejected even though it was a documented argument.

All three are fixed with regression coverage. Focused AddressSanitizer and UndefinedBehaviorSanitizer runs pass. The complete C++20, C++23, C, and Python tests pass. Every current GitHub check is green across Linux, macOS, Windows, WebAssembly, and the QEMU architecture matrix.

The public batch runner also writes a canonical result file. StringZilla, RapidFuzz, and the exact SymSpell compatibility runner produced the same query ID, original dictionary ID, and distance records at k=1 and k=2. The StringZilla and RapidFuzz files also match through k=10 in the separate wide-bound sweep.

Current performance evidence

StringWars #10 contains the query generator, public runners, exact result format, and comparison rules. RapidFuzz is the correctness oracle, not the closest indexed competitor. SymSpell is directly comparable only through its exact compatibility mode on unique lowercase dictionaries. FST, Tantivy, Lucene, and native SymSpell return different information and are reported separately.

The latest local check used 213,557 unique English words, 10,000 mixed queries, one pinned Intel Core i5-9300H AVX2 core, and the exact PR head above. Five warm, pre-sized repetitions measured 6.50 ms at k=1 and 87.91 ms at k=2. The exact-output SymSpell runner measured 65.44 and 718.72 ms. The complete result files matched.

Those numbers are preliminary. The shared host had unrelated load, runner order was not randomized, and later StringZilla-only repetitions varied materially at k=2. They support the architecture and show a strong indexed result, but they are not yet the final speedup claim.

A separate 100-query crossover stress test found StringZilla faster than the exact-output RapidFuzz scan at every bound through k=10. The advantage fell from 10.1x at k=3 to 1.4x at k=10, where 10.6 million of 21.4 million possible pairs matched and result writing dominated. This is evidence that the wide-bound fallback removes the earlier performance cliff, not evidence that k=10 is a normal bound for short English words.

Why this PR is still a draft

There is no known correctness blocker and the project CI is green. It remains a draft for two reasons:

• The API and its boundaries should be agreed before treating a new immutable index as a settled public surface.
• The revised StringWars protocol still needs its final controlled run: at least 20 measured repetitions, randomized runner order, quiet pinned CPUs, raw outputs, memory measurements, and separate AVX2 and AVX-512 evidence.

The intended claim is deliberately narrow: exact CPU retrieval of every original dictionary entry and distance inside a small Levenshtein bound. It is not a claim to be the fastest fuzzy-search library for every result contract, language, bound, or workload.

Follow-ups

• Add nearest and top-N search, with unbounded search as an explicit separate contract.
• Add persistence and evaluate updates through a separate mutable index.
• Revisit a per-length automatic planner if real dictionaries show that the current whole-dictionary estimate leaves useful mixed plans unused.
• Evaluate SIMD dense scanning only after the portable crossover path has controlled evidence across architectures.

Suggested review order

• Commits 1 and 2 establish owned dictionary IDs, exact lookup, and bounded deletion filtering.
• Commits 3 through 6 add the compressed prefix tree, cached distance steps, and automatic construction policy.
• Commits 7 and 8 add UTF-8 behavior and document the retained search paths.
• Commits 9 and 10 expose the C and Python APIs. Commit 11 adds the exhaustive suite.
• Commits 12 through 16 contain the failure-path fixes, behavior notes, and low-bound query optimization.
• Commits 17 and 18 add the wide-bound crossover path and its boundary coverage.