Repository navigation
IndexNode 0.13.2
Per-video descriptor, one append per decision, one file per source, and drama groups.
index/videos/<id>/video.json: one small JSON per upload holds every fact about it that is not job state.v,id;youtube(title, channel, channelId, uploadDate, durationSeconds, url, metadata language);download(today's downloaded.json fields);precheck(the whole record precheck.json held, decision included);speech(language, dubbed, sourceprecheck|transcribe);ai(score, flag, sourceprecheck|centers). Not in it: job state and trash marks (the queue log), groups, classify's record. It is a loose blob, never zipped.- Only the holder of the video's lease writes it. The download creates it (If-None-Match), the process claim updates it (the pre-check record with its speech and AI facts, then the speech language after transcribe, then the AI score from the centers when the pre-check gave none). Every write is conditional on the ETag it read; a 412 reads again, merges again, retries.
- precheck.json is no longer written (the
indexnode precheckcommand still writes one into its local output folder). downloaded.json inside download.zip and theidx_summarymetadata are still written, for Medusa and other readers. - Every reader moved to video.json: the catalog build (one video.json per upload; complete entries of the previous version are reused; no HEAD, precheck.json, transcribe.zip or AI-probe read), the pre-check's matched-upload facts, the group build's short-circuit join,
ProcessStage's and the pre-check's duration, and the search index's speech language. A missing or unreadable video.json fails loudly: the catalog and group builds name the uploads and fail (retried at the next build), a process claim fails and is retried. The duration read that swallowed its errors (MatchedDuration) is gone. indexnode videos backfill [--dry-run] [--threads N]builds video.json for every existing upload from idx_summary (or downloaded.json), info.json, precheck.json, transcribe.zip and the AI probe on centers.zip. Create-only (If-None-Match, so a video.json a claim writes meanwhile is never touched); a video under an active claim (a live lease injobs/active/) is skipped and reported (its claim writes it); an upload with no duration anywhere is reported FAILED, not given an empty descriptor. Run it once, from a westus2 node, before the 0.13.2 search-index job runs: the catalog and group builds need video.json.- One append per decision, not one per line. A refresh used to make one conditional append per new video (live: 43,346 log lines in 43,204 append blocks; about 5,500 appends per 10 minutes and a segment rollover every 3.4 minutes while 570 channel first-lookups ran). Now a lookup decides every new video against the queue state in one
AppendLinesAsync(the unclassified download entries and the classify singleton), and followed channels are added the same way. Blocks are split only by size (JobQueue.Chunks: at most 1000 entries and about 3 MiB;AppendAsyncrefuses a block over Azure's 4 MiB and any line overMaxLineBytes); a 412 decides the whole batch again. Also batched: classify's results (flushed every 500 lines, was 40) and its candidate adds, andtrash empty'sdeletedmarks. Test: a lookup of 200 new videos makes at most 2 AppendBlock calls and leaves the same queue state as one line each. (The refresh's own done line is still its claim's separate final append: the claim fences and ends there.) index/sources/<id>/members.jsonis the one file of a refresh source; seen.json is gone. Each member keeps id, title, duration, channel, availability andfirstSeen. "Is this new?" is "not in members" (no 500-id cap). Written only by the refresh claim, conditional on the ETag read; a playlist is rewritten in full, a channel merges its newest listing into the file, newest first. Migration: a refresh adds the ids of an old seen.json (and any the job body still carries) as id-only members withfirstSeenfrom the file's time, then deletes it, on first touch;indexnode sources migrate-seen [--dry-run]does all sources at once (jobs migratenow writes members directly).
Drama groups: the groups the owner asked for, which PreCheck.cs said did not exist ("per upload today; no group index").
- The pre-check proposes a group membership from its per-frame matches. It joins the group with the largest share of matched frames when that group has at least 5 distinct-frame hits (
precheck.group-min-hits). MIXED = the hits cover one contiguous part of the sample, under 60% of the samples, at a consistent time offset: flagged, never canonical, and it starts its own group. Otherwise NEW. precheck.json getsgroup(kind, join upload, the group's known copies),groupId,groupOffset(seconds),matchShare(frames with cosine >= 0.75 in the group) anddupShare(>= 0.85); the same four, plusmixed, are on the finaldoneline (grp,goff,gms,gds,gmix), duplicates included. - The search-index job is the only writer of groups. Building vN+1 it applies the new uploads' proposed memberships, writes
index/groups/vN+1.json(per group: members with offset, both shares, speechLanguage, dubbed, mixed; canonical and why) and stamps each upload'sgroupinto vN+1's uploads.yaml. A proposed join to an upload that has since moved follows that upload's current group. Old group versions stay, like the old index versions. - Canonical election runs whenever a group changes: Chinese audio only (no language counts as zh), then longest, then first processed (then id); a mixed cut is never canonical; a group of only dubs has no canonical until a Chinese copy arrives. The reason is stored with it (
canonicalWhy). - The pre-check's dub rule uses the group: an original-language upload is processed as canonical when every known copy in the proposed group is a dub, not just the best match. With no group (an index from before 0.13.2) the best match decides as before.
indexnode groups build [--dry-run] [--threads N] [--link 0.75]: the one-time build over the indexed uploads from center-level shares (an upload pair's share of centers whose best center in the other has cosine >= 0.80; a pair links at 0.75), union into groups.--dry-runprints counts and the largest groups and writes nothing; the real run writesindex/groups/vN.jsononce and refuses to write over it.- Not changed: the real store was not touched here, so no first-build time or group counts are reported yet; run
groups build --dry-runon it first. - Fix (live 0.13.1): migration no longer marks a step done that never ran. A step is done only if its old job was done or its output zip is on the store (centers jobs only existed after sscd, so 218 subjects had
centerswrongly done). - Fix: the pre-check gate is decided from the store only (no processing zip besides download/comments and no video.json pre-check record), never from the done set. A step in the done list whose zip is missing runs again, and the final
done processis refused (transient, retried) while any standard step's output is missing (one HEAD each, never a hydrate). - New
indexnode jobs repair-steps [--dry-run]: removes steps whose output is not on the store from stepsDone as batchedfixstepscorrection lines, re-opens finished subjects lacking an output with only the missing steps (no pre-check, search-index follow-up after centers as usual). The dry run prints subjects fixed, steps removed, subjects re-opened and per-step counts. - Fix: default tabs are videos only (Shorts are never dramas). New
indexnode sources set-tabs videos --all [--dry-run](batched lines); a missing tab ("does not have a shorts tab") is an empty listing, not a refresh failure. - Fix: a due search-index entry is claimed ahead of drama work (it was starved for 67 min behind ~200 older priority-0 process entries). Display only: the startup
stages=line omits search-index (it is run by any node with process or a standard step) and a missing space beforelanes=is fixed.