Repository navigation
IndexNode 0.13.9
Lookups read only index/search/; centers.zip is a processing archive. Replaces the "member shards" plan (owner, 2026-11-21).
- Frames file.
index/search/frames/<id>.bin, one per drama, written by the centers step in the same claim (create, a re-run replaces it under the drama's lease). It is the centers.bin layout (header with k/dim/frames/input hash and the drama's sequence number, 64-byte-code-ready center vectors, the center table, then each center's f16 vectors, times and row numbers together) under the magicIXFR, so one ranged read fetches one center. Same f16 values in the same order: exact-cosine results are bit-identical to reading centers.bin. - Readers. The search-index job, the pre-check (indexed and fresh uploads),
/v1/search, the groups full check and the catalog's AI score read frames files; none opens centers.zip (guard testNo_lookup_or_index_build_code_opens_centers_zip). - Index versions (uploads.yaml) record each drama's
fileandtableOff(plus its size, ETag and sha) instead of centers.zip offsets; a lookup skips the HEAD and the header read. A version written before 0.13.9 has nofile: readers open the frames file by name. indexnode frames convert [--dry-run] [--threads N] [--limit N]. Writes frames/.bin for every drama that has a centers.zip and no frames file, from one read of the centers.bin span; create-only, skips dramas under an active claim, the dry run reports the counts. Run once from a westus2 node before the first 0.13.9 index job (an index job with no frames files finds no dramas).- Ghosting.
index/search/is already in.ghostignore.json; after the conversion the 1,129 per-file centers.zip prefixes come out on the machines that have the file, and centers.zip is ghosted like any other step zip (the file is not in this repository). - Local frames cache (owner, from the 0.13.8 live dry run: reads were 94% of the time, ~4,000 range requests and 77 MB per upload, 87 GB over 1,132 uploads, a 2 GB block cache thrashing over ~7.7 GB of frames). A node keeps WHOLE frames files on local disk: each
frames/<id>.binis fetched once with one GET, kept across claims and runs in<config>/frames-cache/as a size-bounded LRU (configindex.local-cache-gb, default 20, 0 = off and range requests as before). Every lookup (pre-check, fresh uploads, /v1/search, the groups check, the index build) reads that local copy by plain file reads: no range request goes to the store. A file is named by drama, input hash and size, so a rewritten frames file is a new local file and the old one ages out. Concurrent callers share one download. Only head/bits/refs and files the node does not have yet go to the store. The 0.13.10 group step gets it for free. - Benchmark (what was measured, and what was not). On a local fake store (12 dramas x 1,500 frames, a 60-frame group check of one drama against the other 11): cold 0.09 s (11 GETs, 19.5 MB), warm 0.042 s, 0 range requests (
Benchmark_cold_download_and_warm_lookups_of_a_drama_group_check). That proves the request count (4,000 per upload down to 0 warm) but NOT the real-store numbers: the cold-start download of ~7.7 GB and the warm group step on the real corpus still have to be timed on a westus2 node afterframes convert(look atframes-cache/growth and the groups progress line). - Tests.
FramesFileTests: frames file = centers.bin bytes after magic/seq; every center block identical from both; index build and search answer the same with every centers.zip deleted and never touch one; convert dry run / create-only / claimed skipped / output equals what the centers step wrote; the source guard.