Repository navigation
IndexNode 0.13.10
Per-drama sequencing, the group step and retained vectors (owner, 2026-11-21). On top of 0.13.9 (frames files and the local cache), so every group step reads frames from the node's disk.
- Sequence number. Each drama gets a monotonically increasing
seq, assigned as aseqlog line INSIDE the fenced conditional append of the process claim, before the pre-check runs (so under the drama's lease). The number is max + 1 of the state the append is conditional on: a lost race (412) re-decides on the newer state, two nodes never get the same one, a repeat never renumbers. It is in the queue state and the snapshot (the maximum survives pruning). indexnode seq assign [--dry-run]. Numbers the existing dramas: by the time of their first pre-check (video.jsonprecheck.at), else the time their centers step finished, else their last queue activity; then id. Continues after the highest number; a drama a node numbers meanwhile is left alone. Limitation: the queue records when centers FINISHED (CentersAt), not when it started, so a drama with no pre-check record is ordered by its centers end.- Group step (
group, after process; added by the process claim, or in bulk byindexnode groups enqueue [--dry-run], create-only, in sequence order). It needs only the VECTORS of earlier dramas, not their grouping: it waits (entry back to pending, due in two minutes) only while a lower-numbered drama has NO frames file yet and its process entry is neither final nor AI-held (still on its way to centers). A lower drama that has frames, is stuck or failed in its own group step, or is held never blocks, so a bootstrap's group steps run in parallel on all lanes and one stuck step blocks nobody. It searches its frames (local copy) against ALL earlier dramas: the index restricted to lower numbers, plus every earlier drama the index lacks, by brute force, each from its retained file if it has one, else its frames file; coverage chains (N covered by P whose frame is covered by Q) resolve through the coverage maps at query time. It writesvideo.jsongrouping: the proposal (join / mixed / new, offset, shares), the covered share and the stretches. - Retained vectors. Settings
retain.enabled(on),retain.t(0.90),retain.f(0.8). A drama that joins a group with at leastfof its frames covered at cosine >=tkeeps only its uncovered frames (at least one) inindex/search/retained/<id>.bin, in the frames-file format; the search-index build uses it instead of the frames file (a retained duplicate adds a handful of centers, not a full set). A re-run that no longer retains removes the file. The coverage thresholds are the defaults of the brief: they have NOT been tuned on the screenshot fixture and the 33 pairs (those assets are not in this repository), so measure them (groups build --dry-run --check-pairs, thenretain.t/retain.f) before relying on the retention. - Index fold (fold only). The search-index job only folds
groupingrecords into groups and the catalog (join, mixed, offset, shares); it never reads or searches frames and never runs the full check, however many dramas lack a record (groups.max-new-per-buildno longer applies to it). A drama without a record is left out of the group set, its own group in the catalog, until its group step has run; the next index build places it. The full check stays only inindexnode groups build [--dry-run --check-pairs], reading the local frames cache. Not built yet: publishing pre-check samples for in-flight predecessors (a drama whose predecessor is still processing waits for its frames instead). - Not measured here. The real-corpus cold download (~7.7 GB) and the warm group step (target well under 10 s) need a westus2 node after
frames convert; the 0.13.9 local benchmark only proves the request count. - Tests.
GroupStepTests: 50 bootstrap group steps on 8 lanes run in parallel with no waiting and each copy joins an earlier one, a lower drama stuck in its group step blocks nobody, a lower drama mid-processing blocks until its frames exist, the index job folds only (ungrouped without a record, no frames read, cap ignored),