Repository navigation
IndexNode 0.13.3
Four fixes from the live 0.13.2 run.
- Backfill no longer fails on a video with no duration in idx_summary. There is no upper bound on duration anywhere in the read path (idx_summary, downloaded.json, info.json and VideoJson all take any positive double; tests pin 25059 s and 28944.5 s). I could not reproduce the reported failure for the four 6 h+ uploads from code: a valid idx_summary with those durations backfills correctly in tests, so the live cause is not established here (re-run backfill on the store; any that still fail will now fall through to the zip, ffprobe and unknown paths). When idx_summary is
{}(the 8 KB metadata overflow) the backfill reads downloaded.json, then info.json, from inside download.zip by ranged reads (directory at the end, then only that member), then runs ffprobe on the first bytes of the stored video.mp4. If every source fails it writesyoutube.durationSeconds: nullwithdurationSource: "unknown"instead of failing. Every duration now records itsdurationSource. - A process claim with no video.json builds it with the backfill's builder (create-only; the claim holds the video's lease), then carries on, instead of failing and retrying forever.
- One upload no longer blocks the search-index build. A missing or unreadable video.json leaves the upload's catalog entry marked
data: "missing"(no duration, title, AI), names it in aWARNINGline, and counts it on the publish line (N with video.json missing); the centers index and groups carry on (the upload is placed with no hint). The next build fills the entry once video.json exists (a missing entry is never reused). A search-index entry that endedfailedis added again by a node that runs the stage (checked when idle, at most every 15 minutes); by hand:indexnode jobs retry --stage search-index. groups builduses the index build's frame-level full check (each upload's own frames searched against the index, joined at >= 5 distinct-frame hits, mixed and offset as in 0.13.2), so existing and new uploads are grouped by one rule; the center-share link (--link) is gone. Join cycles (A joins B while B joins A) are broken: the earlier processed upload owns the group.--check-pairs <file>(linesidA idB) prints how many listed pairs end up in the same group and why the others do not (different groups, not indexed); with--dry-runnothing is written. Group language comes from video.json as it is now (not the index table's copy): a group's canonical reason no longer says "language unknown" for an upload whose video.json has speech.