Skip to content

Releases

IndexNode 0.13.9

Choose a tag to compare

@github-actions github-actions released this 21 Nov 08:26

Lookups read only index/search/; centers.zip is a processing archive. Replaces the "member shards" plan (owner, 2026-11-21).

  • Frames file. index/search/frames/<id>.bin, one per drama, written by the centers step in the same claim (create, a re-run replaces it under the drama's lease). It is the centers.bin layout (header with k/dim/frames/input hash and the drama's sequence number, 64-byte-code-ready center vectors, the center table, then each center's f16 vectors, times and row numbers together) under the magic IXFR, so one ranged read fetches one center. Same f16 values in the same order: exact-cosine results are bit-identical to reading centers.bin.
  • Readers. The search-index job, the pre-check (indexed and fresh uploads), /v1/search, the groups full check and the catalog's AI score read frames files; none opens centers.zip (guard test No_lookup_or_index_build_code_opens_centers_zip).
  • Index versions (uploads.yaml) record each drama's file and tableOff (plus its size, ETag and sha) instead of centers.zip offsets; a lookup skips the HEAD and the header read. A version written before 0.13.9 has no file: readers open the frames file by name.
  • indexnode frames convert [--dry-run] [--threads N] [--limit N]. Writes frames/.bin for every drama that has a centers.zip and no frames file, from one read of the centers.bin span; create-only, skips dramas under an active claim, the dry run reports the counts. Run once from a westus2 node before the first 0.13.9 index job (an index job with no frames files finds no dramas).
  • Ghosting. index/search/ is already in .ghostignore.json; after the conversion the 1,129 per-file centers.zip prefixes come out on the machines that have the file, and centers.zip is ghosted like any other step zip (the file is not in this repository).
  • Local frames cache (owner, from the 0.13.8 live dry run: reads were 94% of the time, ~4,000 range requests and 77 MB per upload, 87 GB over 1,132 uploads, a 2 GB block cache thrashing over ~7.7 GB of frames). A node keeps WHOLE frames files on local disk: each frames/<id>.bin is fetched once with one GET, kept across claims and runs in <config>/frames-cache/ as a size-bounded LRU (config index.local-cache-gb, default 20, 0 = off and range requests as before). Every lookup (pre-check, fresh uploads, /v1/search, the groups check, the index build) reads that local copy by plain file reads: no range request goes to the store. A file is named by drama, input hash and size, so a rewritten frames file is a new local file and the old one ages out. Concurrent callers share one download. Only head/bits/refs and files the node does not have yet go to the store. The 0.13.10 group step gets it for free.
  • Benchmark (what was measured, and what was not). On a local fake store (12 dramas x 1,500 frames, a 60-frame group check of one drama against the other 11): cold 0.09 s (11 GETs, 19.5 MB), warm 0.042 s, 0 range requests (Benchmark_cold_download_and_warm_lookups_of_a_drama_group_check). That proves the request count (4,000 per upload down to 0 warm) but NOT the real-store numbers: the cold-start download of ~7.7 GB and the warm group step on the real corpus still have to be timed on a westus2 node after frames convert (look at frames-cache/ growth and the groups progress line).
  • Tests. FramesFileTests: frames file = centers.bin bytes after magic/seq; every center block identical from both; index build and search answer the same with every centers.zip deleted and never touch one; convert dry run / create-only / claimed skipped / output equals what the centers step wrote; the source guard.

IndexNode 0.13.8

Choose a tag to compare

@github-actions github-actions released this 21 Nov 07:42

One ghost-aware reader. A ghosted blob always has its content (committed, as uncommitted blocks behind an empty commit, or in Nexis's snapshot while it transitions), so no read may fail because of a ghost state, and no file is excluded from ghosting to work around it.

  • The retriever is RestStore, the only store that talks to Azure; LocalCacheStore, StageZipStore, RedirectStore, SplitStore and RecordingStore forward to it, so every read (whole, ranged, conditional, HEAD/length, download, zip open) goes through it.
  • Identity is content, not ETag. A blob is the same when its length and idx_sha256 (when present) match. A ranged read pinned to an ETag that a ghost transition changed (412) is looked at, and read on under the new ETag (remembered, so later reads of the old ETag are not refused); a conditional read treats the same content under a new ETag as unchanged. Only a 404 on the base blob, or different content (Mismatch), is an error.
  • States. committed: read it. ghost: hydrate (commit the blocks, no data transfer), then read. transitioning: read the snapshot; when the snapshot has gone, wait for the base blob to settle, hydrate and read. An empty or out-of-range answer is settled and asked again (bounded), so a race with Nexis changing state between our read and our look is not an error.
  • HEAD gives the real length of a ghost or transitioning blob (the recorded ghostd_size); nothing is hydrated by a HEAD.
  • Folded in: GhostSafeRead (0.13.4) is now only the pin, ZipOpen (0.13.5) hydrates by a plain whole read, the snapshot fallback of 0.13.5 is part of the one sender; CastCandidates no longer calls RestStore directly.
  • Tests. GhostRandomTransitionTests: a simulated Nexis flips a blob between committed, transitioning (its snapshot disappearing) and ghost at random moments while 8 readers run every kind of read in parallel (about 2 s per seed); every read must return the right bytes. OneReaderGuardTests: nothing outside the retriever uses a blob client, the snapshot bookkeeping or ghost hydration.
  • Ghostignore. The 1,129 per-file centers.zip prefixes come out of .ghostignore.json on every machine that has them (the file is not in this repository); keep index/jobs/, index/holds/ and index/gpu-ready/.
  • Groups build on a real store (live 0.13.7 dry run: 3.9 uploads/min at ~263% CPU of 8 cores, no faster than 0.13.6). The faster scan and the parallel uploads did not show because the time is in the member-block range reads, which are round trips. Found: groups build did start --threads uploads at once (default = physical cores), but inside each upload the merged ranges of one centers.zip were read one after the other (the upload's own centers, and each frame chunk's member blocks), so most threads sat waiting on the network. Now the merged ranges of a zip go out together, and the bound is on requests in flight, not threads: at most 64 ranged reads of centers.zip are in flight in the whole process (CentersZip.MaxInFlight). Adjacent blocks of the same zip were already coalesced into one ranged read (gap up to 32 KB, up to 4 MB per read); that is unchanged.
  • The groups progress line has the breakdown: groups: checked N of M (r/min, ETA ...); avg per upload: scan X ms, read Y ms (own centers Z ms), score W ms; R requests, B MB; reads in flight n now, peak p; U uploads at once, T threads each. A fake store with 20 ms per read (8 uploads, 216 reads) now takes about 0.8 s instead of about 4.3 s serial, peak 56 reads in flight. No real-store measurement from here: please re-run the dry run and compare the read figure.
  • Member shards are now 0.13.9.

IndexNode 0.13.7

Choose a tag to compare

@github-actions github-actions released this 21 Nov 07:00

Center-scan speedup, and groups build now uses the cores (live 0.13.6 dry run: 5.6 uploads/min at ~240% CPU of 8 cores, 4.76 GB peak).

  • Two-stage scan (scan.mode fast, default). Stage 1: the symmetric popcount distance of the query's sign code to every code (8 x XOR + PopCount of a ulong, unsafe pointers, no bounds checks) keeps the best R = max(scan.rerank (default 10000), 20 x topC) rows (ties by row; the excluded upload's rows never count). Stage 2: the exact asymmetric LUT score of those R only; the top C come from it in the exact scan's order (score desc, row asc). The old scan is kept as scan.mode exact (also env INDEXNODE_SCAN_MODE=exact) to compare live; new config keys scan.mode, scan.rerank.
  • Batched queries. The 240 frames of a pre-check / group-check chunk are scanned in ONE pass over the codes in 2048-code blocks (the 128 KB block stays in cache while every query scores it).
  • groups build / index job check several uploads at once, up to --threads (default = cores) uploads in parallel, each with its share of the threads for its frames. The checks read the index and the unchanged set; placements are applied after, in index order, so the result does not depend on the thread count. Progress lines count completed uploads.
  • Measured on a 4-core CI runner (no 8-core sandbox here, no live store): scan 54-60 ns/(query,center) exact vs 7-8 ns fast single-thread at 280k synthetic centers (about 8x; 6.7 ns in a 240-query batch), 25 ns exact vs 7-9 ns fast wall with the cores busy; at 41k centers 59 vs 25 ns. Top-100 overlap with the exact scan at 280k synthetic centers: 83% at rerank 2000, 93% at 5000, 97.6% at 10000 (the default, owner-approved), 99.4% at 20000 (about 16 ns/(query,center) single-thread at 10000 vs 54-60 exact); the best row's cosine was identical for every query at every rerank. Group joins and the mixed flag were identical in both modes.
  • Tests: fast scan = exact scan when the shortlist covers the table (order and excluded upload included), same rows at any thread count, best row kept; group joins / mixed cut identical in both modes.

IndexNode 0.13.6

Choose a tag to compare

@github-actions github-actions released this 21 Nov 06:02

Live fix: groups build took ~14 h and 14 GB RSS on 1,083 uploads, and the first search-index build would have done the same inline.

  • The full check samples. Each upload's check searches at most groups.check-frames (default 240; 0 = all) of its frames, evenly spaced over the WHOLE upload, by the pre-check's own sampler (PreCheck.SubsampleIndices) and lookup (PreCheckSearch.BestPerFrameAsync). The pre-check samples a window around 50% of the video; the group check spreads over the full length so mixed cuts and the offset see all of it. The join rule is unchanged (>= 5 distinct-frame hits); shares (match, dup, hit share), the contiguity span and the median offset are computed over the sampled frames (the denominator is the sample count, the offset uses each sample's own time).
  • Bounded memory. The checks share one LRU block cache capped at groups.cache-mb (default 2048); an evicted block is read again. Frames are searched in chunks of 30 so a call never holds every block its frames touch.
  • Progress. groups: checked N of M (r/min, ETA hh:mm) every 10 uploads (and at the end).
  • No inline bootstrap. With no groups file the queued search-index job publishes the version with no groups (catalog groupId null) and logs that indexnode groups build must run first. With a groups file it full-checks only the uploads new in this build, at most groups.max-new-per-build (default 100); the rest wait for the next build. The manual indexnode search-index build/update keeps grouping (capped the same way).
  • New config keys: groups.check-frames, groups.cache-mb, groups.max-new-per-build.
  • Tests: GroupSampleTests (sampler, sampled duplicate join + mixed cut, cache cap in bytes, index job without/with a groups file and the cap, progress line).

IndexNode 0.13.5

Choose a tag to compare

@github-actions github-actions released this 21 Nov 04:28

Two read paths still failed on Nexis ghosts after 0.13.4.

  • A vanished snapshot is not a missing blob. A read that went to a Nexis snapshot (the blob was caught mid-dehydrate) and got a 404 used to end as "centers.zip is gone": Nexis deletes that ephemeral snapshot a few minutes after the transition. RestStore now goes back to the base blob, waits for the transition to finish (about 2 minutes, with its retry backoff), hydrates when it is a ghost, and reads on (GetAsync, ranged reads including the If-Match one, HeadAsync, downloads). Only a base blob that 404s is gone.
  • Every zip open hydrates a ghost first. The backfill's download.zip (duration) and transcribe.zip reads, the describe.zip ending backfill and the stage-zip view took the length from a HEAD (0 for a ghost) and failed with "not a zip: the blob is 0 bytes". They now all go through ZipOpen.HeadReadableAsync, like CentersZip: a 0-byte blob tagged ghostd_state=ghost (or with ghostd_size > 0) is hydrated and HEADed again; a 0-byte blob with no ghost tags is still "not a zip".

IndexNode 0.13.4

Choose a tag to compare

@github-actions github-actions released this 21 Nov 03:50

Live fix: pre-checks failed with "centers.zip is gone" after Nexis dehydrated a centers.zip.

  • A Nexis ghost transition never fails a read. Dehydrating or hydrating a blob changes its ETag, not its bytes, but a reader holding the old ETag got a 412 and called the file gone. One helper (GhostSafeRead, used by CentersZip for the pre-check search, the search-index cache and the catalog/groups builders, by BlobZipSource for every stage-zip range read, and by the backfill's video header probe) now treats a 412 or "gone" on a blob that still exists as: hydrate if it is a ghost, HEAD again, check that the length (and idx_sha256 when both sides have one) are unchanged, then read on with the new ETag. Only a real 404 or changed content is an error.
  • The ffprobe backfill test returns early when ffmpeg/ffprobe are not on PATH (CI runners), with no Skip.

IndexNode 0.13.3

Choose a tag to compare

@github-actions github-actions released this 21 Nov 03:15

Four fixes from the live 0.13.2 run.

  • Backfill no longer fails on a video with no duration in idx_summary. There is no upper bound on duration anywhere in the read path (idx_summary, downloaded.json, info.json and VideoJson all take any positive double; tests pin 25059 s and 28944.5 s). I could not reproduce the reported failure for the four 6 h+ uploads from code: a valid idx_summary with those durations backfills correctly in tests, so the live cause is not established here (re-run backfill on the store; any that still fail will now fall through to the zip, ffprobe and unknown paths). When idx_summary is {} (the 8 KB metadata overflow) the backfill reads downloaded.json, then info.json, from inside download.zip by ranged reads (directory at the end, then only that member), then runs ffprobe on the first bytes of the stored video.mp4. If every source fails it writes youtube.durationSeconds: null with durationSource: "unknown" instead of failing. Every duration now records its durationSource.
  • A process claim with no video.json builds it with the backfill's builder (create-only; the claim holds the video's lease), then carries on, instead of failing and retrying forever.
  • One upload no longer blocks the search-index build. A missing or unreadable video.json leaves the upload's catalog entry marked data: "missing" (no duration, title, AI), names it in a WARNING line, and counts it on the publish line (N with video.json missing); the centers index and groups carry on (the upload is placed with no hint). The next build fills the entry once video.json exists (a missing entry is never reused). A search-index entry that ended failed is added again by a node that runs the stage (checked when idle, at most every 15 minutes); by hand: indexnode jobs retry --stage search-index.
  • groups build uses the index build's frame-level full check (each upload's own frames searched against the index, joined at >= 5 distinct-frame hits, mixed and offset as in 0.13.2), so existing and new uploads are grouped by one rule; the center-share link (--link) is gone. Join cycles (A joins B while B joins A) are broken: the earlier processed upload owns the group. --check-pairs <file> (lines idA idB) prints how many listed pairs end up in the same group and why the others do not (different groups, not indexed); with --dry-run nothing is written. Group language comes from video.json as it is now (not the index table's copy): a group's canonical reason no longer says "language unknown" for an upload whose video.json has speech.

IndexNode 0.13.2

Choose a tag to compare

@github-actions github-actions released this 21 Nov 01:47

Per-video descriptor, one append per decision, one file per source, and drama groups.

  • index/videos/<id>/video.json: one small JSON per upload holds every fact about it that is not job state. v, id; youtube (title, channel, channelId, uploadDate, durationSeconds, url, metadata language); download (today's downloaded.json fields); precheck (the whole record precheck.json held, decision included); speech (language, dubbed, source precheck|transcribe); ai (score, flag, source precheck|centers). Not in it: job state and trash marks (the queue log), groups, classify's record. It is a loose blob, never zipped.
  • Only the holder of the video's lease writes it. The download creates it (If-None-Match), the process claim updates it (the pre-check record with its speech and AI facts, then the speech language after transcribe, then the AI score from the centers when the pre-check gave none). Every write is conditional on the ETag it read; a 412 reads again, merges again, retries.
  • precheck.json is no longer written (the indexnode precheck command still writes one into its local output folder). downloaded.json inside download.zip and the idx_summary metadata are still written, for Medusa and other readers.
  • Every reader moved to video.json: the catalog build (one video.json per upload; complete entries of the previous version are reused; no HEAD, precheck.json, transcribe.zip or AI-probe read), the pre-check's matched-upload facts, the group build's short-circuit join, ProcessStage's and the pre-check's duration, and the search index's speech language. A missing or unreadable video.json fails loudly: the catalog and group builds name the uploads and fail (retried at the next build), a process claim fails and is retried. The duration read that swallowed its errors (MatchedDuration) is gone.
  • indexnode videos backfill [--dry-run] [--threads N] builds video.json for every existing upload from idx_summary (or downloaded.json), info.json, precheck.json, transcribe.zip and the AI probe on centers.zip. Create-only (If-None-Match, so a video.json a claim writes meanwhile is never touched); a video under an active claim (a live lease in jobs/active/) is skipped and reported (its claim writes it); an upload with no duration anywhere is reported FAILED, not given an empty descriptor. Run it once, from a westus2 node, before the 0.13.2 search-index job runs: the catalog and group builds need video.json.
  • One append per decision, not one per line. A refresh used to make one conditional append per new video (live: 43,346 log lines in 43,204 append blocks; about 5,500 appends per 10 minutes and a segment rollover every 3.4 minutes while 570 channel first-lookups ran). Now a lookup decides every new video against the queue state in one AppendLinesAsync (the unclassified download entries and the classify singleton), and followed channels are added the same way. Blocks are split only by size (JobQueue.Chunks: at most 1000 entries and about 3 MiB; AppendAsync refuses a block over Azure's 4 MiB and any line over MaxLineBytes); a 412 decides the whole batch again. Also batched: classify's results (flushed every 500 lines, was 40) and its candidate adds, and trash empty's deleted marks. Test: a lookup of 200 new videos makes at most 2 AppendBlock calls and leaves the same queue state as one line each. (The refresh's own done line is still its claim's separate final append: the claim fences and ends there.)
  • index/sources/<id>/members.json is the one file of a refresh source; seen.json is gone. Each member keeps id, title, duration, channel, availability and firstSeen. "Is this new?" is "not in members" (no 500-id cap). Written only by the refresh claim, conditional on the ETag read; a playlist is rewritten in full, a channel merges its newest listing into the file, newest first. Migration: a refresh adds the ids of an old seen.json (and any the job body still carries) as id-only members with firstSeen from the file's time, then deletes it, on first touch; indexnode sources migrate-seen [--dry-run] does all sources at once (jobs migrate now writes members directly).

Drama groups: the groups the owner asked for, which PreCheck.cs said did not exist ("per upload today; no group index").

  • The pre-check proposes a group membership from its per-frame matches. It joins the group with the largest share of matched frames when that group has at least 5 distinct-frame hits (precheck.group-min-hits). MIXED = the hits cover one contiguous part of the sample, under 60% of the samples, at a consistent time offset: flagged, never canonical, and it starts its own group. Otherwise NEW. precheck.json gets group (kind, join upload, the group's known copies), groupId, groupOffset (seconds), matchShare (frames with cosine >= 0.75 in the group) and dupShare (>= 0.85); the same four, plus mixed, are on the final done line (grp, goff, gms, gds, gmix), duplicates included.
  • The search-index job is the only writer of groups. Building vN+1 it applies the new uploads' proposed memberships, writes index/groups/vN+1.json (per group: members with offset, both shares, speechLanguage, dubbed, mixed; canonical and why) and stamps each upload's group into vN+1's uploads.yaml. A proposed join to an upload that has since moved follows that upload's current group. Old group versions stay, like the old index versions.
  • Canonical election runs whenever a group changes: Chinese audio only (no language counts as zh), then longest, then first processed (then id); a mixed cut is never canonical; a group of only dubs has no canonical until a Chinese copy arrives. The reason is stored with it (canonicalWhy).
  • The pre-check's dub rule uses the group: an original-language upload is processed as canonical when every known copy in the proposed group is a dub, not just the best match. With no group (an index from before 0.13.2) the best match decides as before.
  • indexnode groups build [--dry-run] [--threads N] [--link 0.75]: the one-time build over the indexed uploads from center-level shares (an upload pair's share of centers whose best center in the other has cosine >= 0.80; a pair links at 0.75), union into groups. --dry-run prints counts and the largest groups and writes nothing; the real run writes index/groups/vN.json once and refuses to write over it.
  • Not changed: the real store was not touched here, so no first-build time or group counts are reported yet; run groups build --dry-run on it first.
  • Fix (live 0.13.1): migration no longer marks a step done that never ran. A step is done only if its old job was done or its output zip is on the store (centers jobs only existed after sscd, so 218 subjects had centers wrongly done).
  • Fix: the pre-check gate is decided from the store only (no processing zip besides download/comments and no video.json pre-check record), never from the done set. A step in the done list whose zip is missing runs again, and the final done process is refused (transient, retried) while any standard step's output is missing (one HEAD each, never a hydrate).
  • New indexnode jobs repair-steps [--dry-run]: removes steps whose output is not on the store from stepsDone as batched fixsteps correction lines, re-opens finished subjects lacking an output with only the missing steps (no pre-check, search-index follow-up after centers as usual). The dry run prints subjects fixed, steps removed, subjects re-opened and per-step counts.
  • Fix: default tabs are videos only (Shorts are never dramas). New indexnode sources set-tabs videos --all [--dry-run] (batched lines); a missing tab ("does not have a shorts tab") is an empty listing, not a refresh failure.
  • Fix: a due search-index entry is claimed ahead of drama work (it was starved for 67 min behind ~200 older priority-0 process entries). Display only: the startup stages= line omits search-index (it is run by any node with process or a standard step) and a missing space before lanes= is fixed.

IndexNode 0.13.10

Choose a tag to compare

@github-actions github-actions released this 21 Nov 08:33

Per-drama sequencing, the group step and retained vectors (owner, 2026-11-21). On top of 0.13.9 (frames files and the local cache), so every group step reads frames from the node's disk.

  • Sequence number. Each drama gets a monotonically increasing seq, assigned as a seq log line INSIDE the fenced conditional append of the process claim, before the pre-check runs (so under the drama's lease). The number is max + 1 of the state the append is conditional on: a lost race (412) re-decides on the newer state, two nodes never get the same one, a repeat never renumbers. It is in the queue state and the snapshot (the maximum survives pruning).
  • indexnode seq assign [--dry-run]. Numbers the existing dramas: by the time of their first pre-check (video.json precheck.at), else the time their centers step finished, else their last queue activity; then id. Continues after the highest number; a drama a node numbers meanwhile is left alone. Limitation: the queue records when centers FINISHED (CentersAt), not when it started, so a drama with no pre-check record is ordered by its centers end.
  • Group step (group, after process; added by the process claim, or in bulk by indexnode groups enqueue [--dry-run], create-only, in sequence order). It needs only the VECTORS of earlier dramas, not their grouping: it waits (entry back to pending, due in two minutes) only while a lower-numbered drama has NO frames file yet and its process entry is neither final nor AI-held (still on its way to centers). A lower drama that has frames, is stuck or failed in its own group step, or is held never blocks, so a bootstrap's group steps run in parallel on all lanes and one stuck step blocks nobody. It searches its frames (local copy) against ALL earlier dramas: the index restricted to lower numbers, plus every earlier drama the index lacks, by brute force, each from its retained file if it has one, else its frames file; coverage chains (N covered by P whose frame is covered by Q) resolve through the coverage maps at query time. It writes video.json grouping: the proposal (join / mixed / new, offset, shares), the covered share and the stretches.
  • Retained vectors. Settings retain.enabled (on), retain.t (0.90), retain.f (0.8). A drama that joins a group with at least f of its frames covered at cosine >= t keeps only its uncovered frames (at least one) in index/search/retained/<id>.bin, in the frames-file format; the search-index build uses it instead of the frames file (a retained duplicate adds a handful of centers, not a full set). A re-run that no longer retains removes the file. The coverage thresholds are the defaults of the brief: they have NOT been tuned on the screenshot fixture and the 33 pairs (those assets are not in this repository), so measure them (groups build --dry-run --check-pairs, then retain.t / retain.f) before relying on the retention.
  • Index fold (fold only). The search-index job only folds grouping records into groups and the catalog (join, mixed, offset, shares); it never reads or searches frames and never runs the full check, however many dramas lack a record (groups.max-new-per-build no longer applies to it). A drama without a record is left out of the group set, its own group in the catalog, until its group step has run; the next index build places it. The full check stays only in indexnode groups build [--dry-run --check-pairs], reading the local frames cache. Not built yet: publishing pre-check samples for in-flight predecessors (a drama whose predecessor is still processing waits for its frames instead).
  • Not measured here. The real-corpus cold download (~7.7 GB) and the warm group step (target well under 10 s) need a westus2 node after frames convert; the 0.13.9 local benchmark only proves the request count.
  • Tests. GroupStepTests: 50 bootstrap group steps on 8 lanes run in parallel with no waiting and each copy joins an earlier one, a lower drama stuck in its group step blocks nobody, a lower drama mid-processing blocks until its frames exist, the index job folds only (ungrouped without a record, no frames read, cap ignored),

IndexNode 0.13.1

Choose a tag to compare

@github-actions github-actions released this 20 Nov 23:24
  • Pre-check can no longer be skipped silently on a process claim. A missing search index or SSCD model, a missing duration, a failed cut or a failed analysis now fails the claim (transient, retried) with the reason, instead of logging "skipped" and going on to frames with no record.
  • The pre-check decision is on the final done line of a process claim ("pc":"new|partial|canonical", read from precheck.json), PASS included; a duplicate still carries it in its trash/why fields.
  • Claim order on a running node is covered by a test: a pri line appended by another process while a claim runs reorders the next claims (priority, then added, then id).
  • The search index builds itself as a queue job (search-index): a follow-up is due 15 minutes after a build when more centers finished while it ran, a catch-up build runs when the head is behind the centers on the store, and each version records builtAt (the time it started).
  • Old search-index versions are kept 3 back (search.keep-versions, default 3; 0 keeps only the head); older index/search/vN/ are deleted after head.yaml moves, and a reader that gets a 404 re-reads head.yaml once and retries.
  • The catalog (catalog.json.gz, one per search-index version) lists every upload the pre-check has seen, with duration, title, channel, speech language, dubbed, AI score/flag and the pre-check's match (upload, share, offset, cosine). Every entry has four status fields: state (processed | metadata-only), held ("ai" when held, else null), checkedBy (index | precheck) and data (full | precheck: sample estimates, no centers in the index). A held upload that is processed later is replaced by its full entry.
  • AI dramas can be held: the pre-check flags an AI drama and the process claim releases it as held; indexnode control ai-dramas on|off (off: held subjects are not claimed, on: they resume at frames) and indexnode jobs list --held ai (with the score).
  • The matched upload's duration comes from download.zip's idx_summary metadata (HEAD only, never a hydrate), then the catalog; before, the repacked layout had no downloaded.json and every duplicate was "duration unknown: not skipped".
  • Uploads newer than the search index are found by brute force: the pre-check also compares the sample with the centers of uploads whose centers finished after the index's builtAt.
  • A zero-length ranged read no longer sends an invalid range (to=-1): reading an empty range of a blob now returns nothing instead of failing.