Skip to content

IndexNode 0.13.2

Choose a tag to compare

@github-actions github-actions released this 21 Nov 01:47

Per-video descriptor, one append per decision, one file per source, and drama groups.

  • index/videos/<id>/video.json: one small JSON per upload holds every fact about it that is not job state. v, id; youtube (title, channel, channelId, uploadDate, durationSeconds, url, metadata language); download (today's downloaded.json fields); precheck (the whole record precheck.json held, decision included); speech (language, dubbed, source precheck|transcribe); ai (score, flag, source precheck|centers). Not in it: job state and trash marks (the queue log), groups, classify's record. It is a loose blob, never zipped.
  • Only the holder of the video's lease writes it. The download creates it (If-None-Match), the process claim updates it (the pre-check record with its speech and AI facts, then the speech language after transcribe, then the AI score from the centers when the pre-check gave none). Every write is conditional on the ETag it read; a 412 reads again, merges again, retries.
  • precheck.json is no longer written (the indexnode precheck command still writes one into its local output folder). downloaded.json inside download.zip and the idx_summary metadata are still written, for Medusa and other readers.
  • Every reader moved to video.json: the catalog build (one video.json per upload; complete entries of the previous version are reused; no HEAD, precheck.json, transcribe.zip or AI-probe read), the pre-check's matched-upload facts, the group build's short-circuit join, ProcessStage's and the pre-check's duration, and the search index's speech language. A missing or unreadable video.json fails loudly: the catalog and group builds name the uploads and fail (retried at the next build), a process claim fails and is retried. The duration read that swallowed its errors (MatchedDuration) is gone.
  • indexnode videos backfill [--dry-run] [--threads N] builds video.json for every existing upload from idx_summary (or downloaded.json), info.json, precheck.json, transcribe.zip and the AI probe on centers.zip. Create-only (If-None-Match, so a video.json a claim writes meanwhile is never touched); a video under an active claim (a live lease in jobs/active/) is skipped and reported (its claim writes it); an upload with no duration anywhere is reported FAILED, not given an empty descriptor. Run it once, from a westus2 node, before the 0.13.2 search-index job runs: the catalog and group builds need video.json.
  • One append per decision, not one per line. A refresh used to make one conditional append per new video (live: 43,346 log lines in 43,204 append blocks; about 5,500 appends per 10 minutes and a segment rollover every 3.4 minutes while 570 channel first-lookups ran). Now a lookup decides every new video against the queue state in one AppendLinesAsync (the unclassified download entries and the classify singleton), and followed channels are added the same way. Blocks are split only by size (JobQueue.Chunks: at most 1000 entries and about 3 MiB; AppendAsync refuses a block over Azure's 4 MiB and any line over MaxLineBytes); a 412 decides the whole batch again. Also batched: classify's results (flushed every 500 lines, was 40) and its candidate adds, and trash empty's deleted marks. Test: a lookup of 200 new videos makes at most 2 AppendBlock calls and leaves the same queue state as one line each. (The refresh's own done line is still its claim's separate final append: the claim fences and ends there.)
  • index/sources/<id>/members.json is the one file of a refresh source; seen.json is gone. Each member keeps id, title, duration, channel, availability and firstSeen. "Is this new?" is "not in members" (no 500-id cap). Written only by the refresh claim, conditional on the ETag read; a playlist is rewritten in full, a channel merges its newest listing into the file, newest first. Migration: a refresh adds the ids of an old seen.json (and any the job body still carries) as id-only members with firstSeen from the file's time, then deletes it, on first touch; indexnode sources migrate-seen [--dry-run] does all sources at once (jobs migrate now writes members directly).

Drama groups: the groups the owner asked for, which PreCheck.cs said did not exist ("per upload today; no group index").

  • The pre-check proposes a group membership from its per-frame matches. It joins the group with the largest share of matched frames when that group has at least 5 distinct-frame hits (precheck.group-min-hits). MIXED = the hits cover one contiguous part of the sample, under 60% of the samples, at a consistent time offset: flagged, never canonical, and it starts its own group. Otherwise NEW. precheck.json gets group (kind, join upload, the group's known copies), groupId, groupOffset (seconds), matchShare (frames with cosine >= 0.75 in the group) and dupShare (>= 0.85); the same four, plus mixed, are on the final done line (grp, goff, gms, gds, gmix), duplicates included.
  • The search-index job is the only writer of groups. Building vN+1 it applies the new uploads' proposed memberships, writes index/groups/vN+1.json (per group: members with offset, both shares, speechLanguage, dubbed, mixed; canonical and why) and stamps each upload's group into vN+1's uploads.yaml. A proposed join to an upload that has since moved follows that upload's current group. Old group versions stay, like the old index versions.
  • Canonical election runs whenever a group changes: Chinese audio only (no language counts as zh), then longest, then first processed (then id); a mixed cut is never canonical; a group of only dubs has no canonical until a Chinese copy arrives. The reason is stored with it (canonicalWhy).
  • The pre-check's dub rule uses the group: an original-language upload is processed as canonical when every known copy in the proposed group is a dub, not just the best match. With no group (an index from before 0.13.2) the best match decides as before.
  • indexnode groups build [--dry-run] [--threads N] [--link 0.75]: the one-time build over the indexed uploads from center-level shares (an upload pair's share of centers whose best center in the other has cosine >= 0.80; a pair links at 0.75), union into groups. --dry-run prints counts and the largest groups and writes nothing; the real run writes index/groups/vN.json once and refuses to write over it.
  • Not changed: the real store was not touched here, so no first-build time or group counts are reported yet; run groups build --dry-run on it first.
  • Fix (live 0.13.1): migration no longer marks a step done that never ran. A step is done only if its old job was done or its output zip is on the store (centers jobs only existed after sscd, so 218 subjects had centers wrongly done).
  • Fix: the pre-check gate is decided from the store only (no processing zip besides download/comments and no video.json pre-check record), never from the done set. A step in the done list whose zip is missing runs again, and the final done process is refused (transient, retried) while any standard step's output is missing (one HEAD each, never a hydrate).
  • New indexnode jobs repair-steps [--dry-run]: removes steps whose output is not on the store from stepsDone as batched fixsteps correction lines, re-opens finished subjects lacking an output with only the missing steps (no pre-check, search-index follow-up after centers as usual). The dry run prints subjects fixed, steps removed, subjects re-opened and per-step counts.
  • Fix: default tabs are videos only (Shorts are never dramas). New indexnode sources set-tabs videos --all [--dry-run] (batched lines); a missing tab ("does not have a shorts tab") is an empty listing, not a refresh failure.
  • Fix: a due search-index entry is claimed ahead of drama work (it was starved for 67 min behind ~200 older priority-0 process entries). Display only: the startup stages= line omits search-index (it is run by any node with process or a standard step) and a missing space before lanes= is fixed.