Skip to content

feat: index video files as their extracted still frame - #361

Merged
lstein merged 2 commits into
masterfrom
lstein/feature/index-videos
Aug 17, 2026
Merged

feat: index video files as their extracted still frame#361
lstein merged 2 commits into
masterfrom
lstein/feature/index-videos

Conversation

@lstein

@lstein lstein commented Aug 15, 2026

Copy link
Copy Markdown
Owner

PR 7 of 8 — ⬅ the flip. Stacked on #360.

This is the commit that turns video support on. Everything it activates was merged and tested in #355#360, which also makes it the single revertible commit that turns it back off.

The walk now defaults to INDEXABLE_EXTENSIONS, and _load_image dispatches videos to a new _load_video returning the same (frame, modtime, metadata) shape. Everything downstream is therefore media-agnostic — a video is CLIP-encoded, clustered, searchable and curatable exactly like a photo.

The gate bypass is the sharpest thing in this PR

_passes_dimension_gate returns early for videos, and that early return is the point rather than an accident of the byte bands:

  • the middle band opens the file with PIL, which raises on a video
  • the caller memoizes that False into scan_rejects.npz keyed by (size, mtime)
  • those only change if the file itself does — so a video rejected once stays invisible forever, with no UI to clear it

And it would not have shown up in manual testing: most real videos exceed the 500 KB probe ceiling and pass on size alone, so this only bites with small clips. scan_rejects.npz gains a cache_version, bumped here, so anyone who ran an intermediate build gets their cache discarded once.

Batch isolation

flush() now retries a failed batch one item at a time. Extracted frames are the first realistic source of a PIL object the encoder chokes on, and losing seven unrelated photos to one bad video would be confusing and hard to attribute.

Cache lifecycle — one sweep, not seven hooks

Stills are swept at save time, when the index is authoritative. That single sweep covers mtime changes, moves, copies, single and batch deletes, and files removed outside the app. Delete also discards immediately (a stale frame for a deleted file is exactly what users notice), and album deletion clears the album's cache.

Reporting

bad_files were collected but surfaced nowhere — users just saw a smaller count than expected. They now become a completion warning, composed with rather than clobbering the board album's missing-on-disk notice (that's what #360's add_completion_warning was for).

Board albums stay image-only: deletion there routes through invokeai_client.delete_image, unverified against video assets, and indexing something we couldn't then delete would be worse than skipping it.

Tests

19 end-to-end in test_video_indexing.py, against a new new_media_album fixture copying test_images/ and test_media/ — so existing suites keep their exact counts and only video tests pay the ffmpeg cost.

The most valuable one: test_cached_frames_are_not_reindexed_as_photos runs the update twice and asserts a stable count. Stills are full-resolution and sail through both gates, so if the cache ever lands inside a scanned tree every still becomes a photo — and each of those gets a still of its own, forever.

Also covers: the truncated file being skipped without aborting, the gate bypass under an absurd min_image_dimension, videos never entering scan_rejects, the stale-cache discard, prune-on-update, delete-removes-frame, UMAP inclusion, and a deterministic self-match search (a video's own frame is its own top result — no claims about CLIP's view of a test pattern).

Backend 609 passed, frontend 501 passed, ruff clean.

🤖 Generated with Claude Code

@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from 4305572 to a32b9aa Compare August 16, 2026 19:59
@lstein
lstein force-pushed the lstein/feature/index-videos branch from d0dcd55 to e6317ba Compare August 16, 2026 19:59
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from a32b9aa to 2bf3652 Compare August 16, 2026 21:42
@lstein
lstein force-pushed the lstein/feature/index-videos branch from e6317ba to 27da0e6 Compare August 16, 2026 21:43
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from 2bf3652 to ca1a9a4 Compare August 17, 2026 00:33
@lstein
lstein force-pushed the lstein/feature/index-videos branch from 27da0e6 to 34a5fef Compare August 17, 2026 00:33
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from ca1a9a4 to 9b20f94 Compare August 17, 2026 01:56
@lstein
lstein force-pushed the lstein/feature/index-videos branch from 34a5fef to 507b6eb Compare August 17, 2026 01:56
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from 9b20f94 to 935cccc Compare August 17, 2026 02:53
@lstein
lstein force-pushed the lstein/feature/index-videos branch from 507b6eb to 1f61f55 Compare August 17, 2026 02:53
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from 935cccc to dd91950 Compare August 17, 2026 02:58
@lstein
lstein force-pushed the lstein/feature/index-videos branch from 1f61f55 to dc2aed8 Compare August 17, 2026 02:58
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from dd91950 to bd5276a Compare August 17, 2026 03:38
@lstein
lstein force-pushed the lstein/feature/index-videos branch from dc2aed8 to 494fc67 Compare August 17, 2026 03:38
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from bd5276a to 2f85ccf Compare August 17, 2026 03:52
@lstein
lstein force-pushed the lstein/feature/index-videos branch from 494fc67 to df99f07 Compare August 17, 2026 03:52
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from 2f85ccf to 60f945c Compare August 17, 2026 04:06
@lstein
lstein force-pushed the lstein/feature/index-videos branch from df99f07 to aafcb15 Compare August 17, 2026 04:06
@lstein
lstein force-pushed the lstein/feature/video-backend-guards branch from 60f945c to 878a453 Compare August 17, 2026 10:42
@lstein
lstein force-pushed the lstein/feature/index-videos branch 2 times, most recently from 32f3510 to ef012f3 Compare August 17, 2026 10:52
Base automatically changed from lstein/feature/video-backend-guards to master August 17, 2026 11:22
This is the commit that turns video support on. Everything it activates was
merged and tested in the preceding PRs, so it is also the single revertible
commit that turns it back off.

The directory walk now defaults to INDEXABLE_EXTENSIONS, and _load_image
dispatches videos to a new _load_video, which extracts a frame, caches it, and
returns the same (frame, modtime, metadata) shape. Everything downstream is
therefore media-agnostic: a video is CLIP-encoded, clustered, searchable and
curatable exactly like a photo. Video facts ride inside the existing per-image
metadata dict, so every .npz rewrite path carries them for free and indexes
predating this need no migration.

_passes_dimension_gate returns early for videos, and that early return is the
point rather than an accident of the byte bands. The gate's middle band opens
the file with PIL, which raises on a video, and the caller memoizes that
rejection into scan_rejects.npz keyed by (size, mtime) — which only changes if
the file does. A video rejected once would stay invisible forever with no UI
to clear it. Most real videos exceed the 500 KB probe ceiling and pass on size
alone, so this would never have reproduced in manual testing with real
footage, only with small clips. scan_rejects.npz gains a cache_version, bumped
here, so anyone who ran an intermediate build has their cache discarded once.

flush() now retries a failed batch one item at a time. Extracted frames are
the first realistic source of a PIL object the encoder chokes on, and losing
seven unrelated photos to one bad video would be confusing and hard to
attribute.

Cached stills are swept at save time, when the index is authoritative. One
sweep covers what would otherwise need seven hooks: mtime changes, moves,
copies, single and batch deletes, and files removed outside the app. Delete
also discards immediately, since a stale frame for a deleted file is exactly
what users notice, and album deletion clears the album's cache.

bad_files were collected but reported nowhere — the user just saw a smaller
count than expected. They now surface as a completion warning, composed with
rather than clobbering the board album's missing-on-disk notice.

Board albums stay image-only: deletion there routes through
invokeai_client.delete_image, which is unverified against video assets, and
indexing something we could not then delete would be worse than skipping it.

Tests: 19 end-to-end in test_video_indexing.py, against a new_media_album
fixture that copies test_images/ AND test_media/ so existing suites keep their
exact counts. Includes the regression that matters most — running the update
twice and asserting a stable count, which is what catches the frame cache
being re-indexed as photos. Backend 609 passed, frontend 501 passed, ruff
clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lstein
lstein force-pushed the lstein/feature/index-videos branch from ef012f3 to ed40595 Compare August 17, 2026 11:23
The "N files were skipped" notice never reached the user. It was
registered from the router after create_index_async/update_index_async
returned — but complete_operation runs *inside* those calls, and it is
what folds pending notices into the ProgressInfo the poller reads and then
clears the queue. So the notice was stranded: invisible for its own run,
and silently attached to whichever run completed next. Verified directly:

    PENDING DICT: {'test_media_album': ['1 file could not be read...']}
    PROGRESS STATUS: completed
    WARNING FIELD: None

Registration moved into the indexing calls, ahead of every
complete_operation (including the no-new-images path, which can still
have skipped files). Doing it in the router could not be fixed by
re-folding after the fact: the frontend stops polling once it sees
COMPLETED, so a notice attached later races the last poll.

Its test passed only through cross-test pollution. It read the previous
test's stranded warning — it fails when run on its own, and passes again
as soon as any other indexing test precedes it in the same process. The
suite now clears the shared progress_tracker queue around each test, the
assertion pins the exact message, and a clean album is asserted to report
no notice at all as a control.

Also "1 file could not be read and were skipped" — the noun was
pluralised, the verb was not.

The dimension-gate test only exercised one of the two bands. It set
min_image_bytes=0, disabling the byte floor, so it would have passed with
the early return moved below the floor check. That matters: clip.mp4 is
~2 KB and clip.webm ~940 bytes, both under the *default* min_image_bytes
of 8192, so small real videos would be rejected on size alone. The test
now raises both bands, and a separate case pins the default floor.

Confirmed all of these fail without the fix: removing the early return
breaks 10 tests in the file.

Test-only: index into "results" in test_search_still_accepts_an_image_
query. The response is a SearchResultsResponse, so len() over the whole
payload counted its single key and was always truthy.
@lstein
lstein merged commit b6a78f5 into master Aug 17, 2026
9 checks passed
@lstein
lstein deleted the lstein/feature/index-videos branch August 17, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant