Skip to content

lexindex v4.1.0

Choose a tag to compare

@github-actions github-actions released this 22 Sep 19:37
· 268 commits to main since this release
v4.1.0
797b601

Added

  • HashedDictIndex: a DictIndex whose id is a hash. A dictionary answers id by searching
    — the block samples, one block's restarts, one microblock's scan — and that search is the column
    the frontier campaign lost: XCDAT 1.2–5.1× faster at a million keys, and 1.1–1.8× at ten million
    on five of the six corpora there.
    HashedDictIndex::from_dict(dict, fingerprint_bits) keeps the dictionary whole and adds a minimal
    perfect hash over the keys' 64-bit hashes and a bit-packed table holding each key's rank at its
    slot, ⌈log2 n⌉ + fingerprint_bits bits wide. id is two hashes of the key and two reads and
    never touches the dictionary; key(id), prefix, range and iteration go to dict(), whose ids are
    the same ranks. On the campaign's protocol, beside Dict 256 and XCDAT 15 as controls that read
    within 4.9 % of the published campaign on all 38 of their cells, it is faster than XCDAT and
    smaller on all nineteen corpora
    : id_unchecked 3.9–8.9× faster at 1.39–3.16× less space, id
    at eight fingerprint bits 3.1–7.6× and 1.17–2.77×, and it builds before XCDAT on every one.
    Against its own dictionary's search it is 7.6–20.5× faster closed and 6.1–17.0× at eight bits,
    for 2.62 bytes a key on the 480 k-word dictionary, 2.74 at a million keys and 3.24 at ten
    million.

    The width is chosen once, at build, and it decides the contract. From one fingerprint bit up id
    answers a stranger as present at 2^-fingerprint_bits, CompactHashIndex's rate; at zero bits
    id is the dictionary's exact search, and the hash is id_unchecked alone — a member's rank, and
    some rank below n for anything else. Keys sharing a 64-bit hash are told apart by the full
    second hash in a side table, so a member's rank is exact at every width. The format is BHD1: a
    48-byte header, the dictionary's BDX3 blob byte for byte — checked by DictIndex's own loader —
    then the perfect hash, the rank table and the side table. load_mmap borrows the dictionary and
    the rank table, and a crafted blob answers wrong ranks, never ranks past the end. Python has the
    same class, HashedDictIndex.from_dict(dict, fingerprint_bits=8), with ids_of, in,
    indexing and .dict, and the batch forms DictIndex has — ids_of_bytes, ids_into and the
    Arrow pair ids_of_arrow and ids_into_arrow — writing uint64 ids and MISSING_ID for a
    miss. The C ABI opens BHD1 blobs and builds the index at eight bits as
    LEXINDEX_KIND_HASHED_DICT, a sixth kind under the same ABI version: a caller compiled against
    the older header meets it only for a blob its library refused. plan does not price it yet, and
    Overlay does not take it: its ids are ranks.

  • The command line in the wheel. plan, build, inspect and dump were reachable only
    through cargo install lexindex, so a pip install could price a corpus from Python code but not
    from a shell. The wheel now carries the binary's own program — src/cli.rs, compiled into both —
    as a lexindex command and as python -m lexindex, so the subcommands, their output and their
    exit statuses are the same whichever way lexindex was installed:
    pip install lexindex && lexindex plan keys.txt. Ctrl-C ends a run as it ends the binary's,
    rather than waiting for the Rust code to return.

  • A Polars plugin, in polars/. lexindex-polars is a package of its own — its own crate,
    wheel, version and changelog — carrying four expressions over an index blob named by path:
    pl.col("track").lexindex.id("tracks.bdx"), and id_unchecked, contains and key beside it.
    They are plugin expressions rather than a map_batches callback, so polars runs them in its own
    threads, inside a lazy plan and under the streaming engine, with no GIL; one expression serves all
    six index kinds, because the blob says which kind it is. Nothing about the lexindex crate or
    wheel changes: polars' plugin ABI moves on polars' cadence, which is not this library's, so it
    stays out of the dependency tree and ships separately.

  • A fuzz target for BHD1. parse_hashed_dict loads arbitrary bytes as a HashedDictIndex
    and holds every rank id, id_unchecked and ids_of answer below the key count; it runs in CI's
    fuzz matrix beside parse_dict, and tests/data/golden-4.1.0-hashed.bhd pins the format by its
    bytes and seeds the corpus.

  • Pull requests are fuzzed. One that touches src/, fuzz/ or the seeds runs every libFuzzer
    target, ten minutes between them, under ClusterFuzzLite — OSS-Fuzz's build and fuzzing images, on
    the repository's own runners — before it is merged; the daily campaign still fuzzes main.
    .clusterfuzzlite/ is the build an OSS-Fuzz project would use, and it builds all ten targets with
    the instrumentation intact: 2 448 counters in parse_compact, where an LTO build keeps a few
    hundred.

  • The C ABI under the sanitizers. The weekly AddressSanitizer run compiled tests/capi.rs to
    nothing, since it needs the capi feature, and Miri could not run it at all: the test calls the
    C functions through copies of the header's #[repr(C)] enums, which Miri holds to be other types
    than the callee's and stops at the first call. Both run it every week now, Miri through the
    module's own types, and both pass.

  • bench/frontier/run.sh --only PATTERN runs the campaign's processes whose label matches,
    into an artifact named frontier-<scale>-subset-*; frontier_lex takes hashed0, hashed8 and
    hashed16, and bench/tables.py renders a campaign's HashedDictIndex table with view=hashed.

Fixed

  • bench/frontier/run.sh refused to run at 4.0.1: it builds frontier_lex with --locked, and
    that crate's lock file still named lexindex 4.0.0 — as bench/mphf_vs's did. CI now fails when a
    tracked Cargo.lock names another version of lexindex than Cargo.toml does, so a release
    commit that misses one is red before it is tagged.
  • The release preflight checks the tag's own date. It compared CITATION.cff's
    date-released with the CHANGELOG heading and neither with the day of the tag, so a release
    prepared on one day and tagged on another passed — 4.0.0 was re-dated twice by hand. It now reads
    the annotated tag's date, in the tagger's zone, and refuses a mismatch or a lightweight tag before
    anything is built.

Documentation

  • docs/api.md rendered four of the public names. Overlay (public since 0.12), DictIndex
    (2.0), plan (3.0), inspect and the types they return were never on it; every name in
    __all__ is now, and a test fails when one is not.
  • CI fails when a page cites a bench/results file that is not committed. A citation of an
    untracked artifact is a 404 from the day it is pushed; one reached a commit this cycle and was
    caught by hand before the push.
  • plan_file's page on docs.rs has linked a private constant since 4.0.0, a dead link there, and
    HashedDictIndex's contract — what fingerprint_bits decides — sat in a private module's
    documentation, which docs.rs does not render. Both are on the public pages now, and CI denies
    rustdoc's warnings on the build docs.rs makes, where a link to a private item is reported.
  • SECURITY.md listed the fuzzed formats as of 1.1, without MPH3, the perfect hash's format since
    4.0.0; it names MPH3 and BHD1 now. Its unsafe inventory is recounted from the source — 15
    unsafe fns where it said 13, 41 blocks where it said 38, eleven in the Python bindings where it
    said nine, HashedDictIndex's memory maps and the phrase trie's unchecked walk among the
    difference — and names what it left out: the four unsafe impls of Send and Sync, the
    unsafe trait behind zeroed huge-page tables, and the C ABI's five internal unsafe fns.