Skip to content

IndexNode 0.13.10

Choose a tag to compare

@github-actions github-actions released this 21 Nov 08:33

Per-drama sequencing, the group step and retained vectors (owner, 2026-11-21). On top of 0.13.9 (frames files and the local cache), so every group step reads frames from the node's disk.

  • Sequence number. Each drama gets a monotonically increasing seq, assigned as a seq log line INSIDE the fenced conditional append of the process claim, before the pre-check runs (so under the drama's lease). The number is max + 1 of the state the append is conditional on: a lost race (412) re-decides on the newer state, two nodes never get the same one, a repeat never renumbers. It is in the queue state and the snapshot (the maximum survives pruning).
  • indexnode seq assign [--dry-run]. Numbers the existing dramas: by the time of their first pre-check (video.json precheck.at), else the time their centers step finished, else their last queue activity; then id. Continues after the highest number; a drama a node numbers meanwhile is left alone. Limitation: the queue records when centers FINISHED (CentersAt), not when it started, so a drama with no pre-check record is ordered by its centers end.
  • Group step (group, after process; added by the process claim, or in bulk by indexnode groups enqueue [--dry-run], create-only, in sequence order). It needs only the VECTORS of earlier dramas, not their grouping: it waits (entry back to pending, due in two minutes) only while a lower-numbered drama has NO frames file yet and its process entry is neither final nor AI-held (still on its way to centers). A lower drama that has frames, is stuck or failed in its own group step, or is held never blocks, so a bootstrap's group steps run in parallel on all lanes and one stuck step blocks nobody. It searches its frames (local copy) against ALL earlier dramas: the index restricted to lower numbers, plus every earlier drama the index lacks, by brute force, each from its retained file if it has one, else its frames file; coverage chains (N covered by P whose frame is covered by Q) resolve through the coverage maps at query time. It writes video.json grouping: the proposal (join / mixed / new, offset, shares), the covered share and the stretches.
  • Retained vectors. Settings retain.enabled (on), retain.t (0.90), retain.f (0.8). A drama that joins a group with at least f of its frames covered at cosine >= t keeps only its uncovered frames (at least one) in index/search/retained/<id>.bin, in the frames-file format; the search-index build uses it instead of the frames file (a retained duplicate adds a handful of centers, not a full set). A re-run that no longer retains removes the file. The coverage thresholds are the defaults of the brief: they have NOT been tuned on the screenshot fixture and the 33 pairs (those assets are not in this repository), so measure them (groups build --dry-run --check-pairs, then retain.t / retain.f) before relying on the retention.
  • Index fold (fold only). The search-index job only folds grouping records into groups and the catalog (join, mixed, offset, shares); it never reads or searches frames and never runs the full check, however many dramas lack a record (groups.max-new-per-build no longer applies to it). A drama without a record is left out of the group set, its own group in the catalog, until its group step has run; the next index build places it. The full check stays only in indexnode groups build [--dry-run --check-pairs], reading the local frames cache. Not built yet: publishing pre-check samples for in-flight predecessors (a drama whose predecessor is still processing waits for its frames instead).
  • Not measured here. The real-corpus cold download (~7.7 GB) and the warm group step (target well under 10 s) need a westus2 node after frames convert; the 0.13.9 local benchmark only proves the request count.
  • Tests. GroupStepTests: 50 bootstrap group steps on 8 lanes run in parallel with no waiting and each copy joins an earlier one, a lower drama stuck in its group step blocks nobody, a lower drama mid-processing blocks until its frames exist, the index job folds only (ungrouped without a record, no frames read, cap ignored),