feat: index video files as their extracted still frame - #361
Merged
Conversation
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 16, 2026 19:59
4305572 to
a32b9aa
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 16, 2026 19:59
d0dcd55 to
e6317ba
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 16, 2026 21:42
a32b9aa to
2bf3652
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 16, 2026 21:43
e6317ba to
27da0e6
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 00:33
2bf3652 to
ca1a9a4
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 00:33
27da0e6 to
34a5fef
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 01:56
ca1a9a4 to
9b20f94
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 01:56
34a5fef to
507b6eb
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 02:53
9b20f94 to
935cccc
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 02:53
507b6eb to
1f61f55
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 02:58
935cccc to
dd91950
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 02:58
1f61f55 to
dc2aed8
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 03:38
dd91950 to
bd5276a
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 03:38
dc2aed8 to
494fc67
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 03:52
bd5276a to
2f85ccf
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 03:52
494fc67 to
df99f07
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 04:06
2f85ccf to
60f945c
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 04:06
df99f07 to
aafcb15
Compare
lstein
force-pushed
the
lstein/feature/video-backend-guards
branch
from
August 17, 2026 10:42
60f945c to
878a453
Compare
lstein
force-pushed
the
lstein/feature/index-videos
branch
2 times, most recently
from
August 17, 2026 10:52
32f3510 to
ef012f3
Compare
This is the commit that turns video support on. Everything it activates was merged and tested in the preceding PRs, so it is also the single revertible commit that turns it back off. The directory walk now defaults to INDEXABLE_EXTENSIONS, and _load_image dispatches videos to a new _load_video, which extracts a frame, caches it, and returns the same (frame, modtime, metadata) shape. Everything downstream is therefore media-agnostic: a video is CLIP-encoded, clustered, searchable and curatable exactly like a photo. Video facts ride inside the existing per-image metadata dict, so every .npz rewrite path carries them for free and indexes predating this need no migration. _passes_dimension_gate returns early for videos, and that early return is the point rather than an accident of the byte bands. The gate's middle band opens the file with PIL, which raises on a video, and the caller memoizes that rejection into scan_rejects.npz keyed by (size, mtime) — which only changes if the file does. A video rejected once would stay invisible forever with no UI to clear it. Most real videos exceed the 500 KB probe ceiling and pass on size alone, so this would never have reproduced in manual testing with real footage, only with small clips. scan_rejects.npz gains a cache_version, bumped here, so anyone who ran an intermediate build has their cache discarded once. flush() now retries a failed batch one item at a time. Extracted frames are the first realistic source of a PIL object the encoder chokes on, and losing seven unrelated photos to one bad video would be confusing and hard to attribute. Cached stills are swept at save time, when the index is authoritative. One sweep covers what would otherwise need seven hooks: mtime changes, moves, copies, single and batch deletes, and files removed outside the app. Delete also discards immediately, since a stale frame for a deleted file is exactly what users notice, and album deletion clears the album's cache. bad_files were collected but reported nowhere — the user just saw a smaller count than expected. They now surface as a completion warning, composed with rather than clobbering the board album's missing-on-disk notice. Board albums stay image-only: deletion there routes through invokeai_client.delete_image, which is unverified against video assets, and indexing something we could not then delete would be worse than skipping it. Tests: 19 end-to-end in test_video_indexing.py, against a new_media_album fixture that copies test_images/ AND test_media/ so existing suites keep their exact counts. Includes the regression that matters most — running the update twice and asserting a stable count, which is what catches the frame cache being re-indexed as photos. Backend 609 passed, frontend 501 passed, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lstein
force-pushed
the
lstein/feature/index-videos
branch
from
August 17, 2026 11:23
ef012f3 to
ed40595
Compare
The "N files were skipped" notice never reached the user. It was
registered from the router after create_index_async/update_index_async
returned — but complete_operation runs *inside* those calls, and it is
what folds pending notices into the ProgressInfo the poller reads and then
clears the queue. So the notice was stranded: invisible for its own run,
and silently attached to whichever run completed next. Verified directly:
PENDING DICT: {'test_media_album': ['1 file could not be read...']}
PROGRESS STATUS: completed
WARNING FIELD: None
Registration moved into the indexing calls, ahead of every
complete_operation (including the no-new-images path, which can still
have skipped files). Doing it in the router could not be fixed by
re-folding after the fact: the frontend stops polling once it sees
COMPLETED, so a notice attached later races the last poll.
Its test passed only through cross-test pollution. It read the previous
test's stranded warning — it fails when run on its own, and passes again
as soon as any other indexing test precedes it in the same process. The
suite now clears the shared progress_tracker queue around each test, the
assertion pins the exact message, and a clean album is asserted to report
no notice at all as a control.
Also "1 file could not be read and were skipped" — the noun was
pluralised, the verb was not.
The dimension-gate test only exercised one of the two bands. It set
min_image_bytes=0, disabling the byte floor, so it would have passed with
the early return moved below the floor check. That matters: clip.mp4 is
~2 KB and clip.webm ~940 bytes, both under the *default* min_image_bytes
of 8192, so small real videos would be rejected on size alone. The test
now raises both bands, and a separate case pins the default floor.
Confirmed all of these fail without the fix: removing the early return
breaks 10 tests in the file.
Test-only: index into "results" in test_search_still_accepts_an_image_
query. The response is a SearchResultsResponse, so len() over the whole
payload counted its single key and was always truthy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 7 of 8 — ⬅ the flip. Stacked on #360.
This is the commit that turns video support on. Everything it activates was merged and tested in #355–#360, which also makes it the single revertible commit that turns it back off.
The walk now defaults to
INDEXABLE_EXTENSIONS, and_load_imagedispatches videos to a new_load_videoreturning the same(frame, modtime, metadata)shape. Everything downstream is therefore media-agnostic — a video is CLIP-encoded, clustered, searchable and curatable exactly like a photo.The gate bypass is the sharpest thing in this PR
_passes_dimension_gatereturns early for videos, and that early return is the point rather than an accident of the byte bands:Falseintoscan_rejects.npzkeyed by(size, mtime)And it would not have shown up in manual testing: most real videos exceed the 500 KB probe ceiling and pass on size alone, so this only bites with small clips.
scan_rejects.npzgains acache_version, bumped here, so anyone who ran an intermediate build gets their cache discarded once.Batch isolation
flush()now retries a failed batch one item at a time. Extracted frames are the first realistic source of a PIL object the encoder chokes on, and losing seven unrelated photos to one bad video would be confusing and hard to attribute.Cache lifecycle — one sweep, not seven hooks
Stills are swept at save time, when the index is authoritative. That single sweep covers mtime changes, moves, copies, single and batch deletes, and files removed outside the app. Delete also discards immediately (a stale frame for a deleted file is exactly what users notice), and album deletion clears the album's cache.
Reporting
bad_fileswere collected but surfaced nowhere — users just saw a smaller count than expected. They now become a completion warning, composed with rather than clobbering the board album's missing-on-disk notice (that's what #360'sadd_completion_warningwas for).Board albums stay image-only: deletion there routes through
invokeai_client.delete_image, unverified against video assets, and indexing something we couldn't then delete would be worse than skipping it.Tests
19 end-to-end in
test_video_indexing.py, against a newnew_media_albumfixture copyingtest_images/andtest_media/— so existing suites keep their exact counts and only video tests pay the ffmpeg cost.The most valuable one:
test_cached_frames_are_not_reindexed_as_photosruns the update twice and asserts a stable count. Stills are full-resolution and sail through both gates, so if the cache ever lands inside a scanned tree every still becomes a photo — and each of those gets a still of its own, forever.Also covers: the truncated file being skipped without aborting, the gate bypass under an absurd
min_image_dimension, videos never enteringscan_rejects, the stale-cache discard, prune-on-update, delete-removes-frame, UMAP inclusion, and a deterministic self-match search (a video's own frame is its own top result — no claims about CLIP's view of a test pattern).Backend 609 passed, frontend 501 passed, ruff clean.
🤖 Generated with Claude Code