feat(kernel): thumbnails cached at ingest, with a backfill for what predates them (#21) - #93
Merged
Merged
Conversation
…redates them (#21) Every ingested item now gets a preview, stored content-addressed beside its content and named by `Asset.thumbnail_hash`. The M5 gallery is the reason: decoding full-resolution images per tile at request time is the wrong shape, and the cost amortizes here, where the bytes are already open. A thumbnail hash is a cache key, not an identity — absent from every release manifest, never recomputed by `ReleaseService.verify`, and legitimately different across Pillow builds. Everything else follows from that: - A preview that will not render is NOT an `IngestFailure`. The asset exists and nothing was lost, so the hash stays NULL and the run carries on with an empty report. The `try` is nested inside the caller's, because reaching the outer `except MediaError` would report a good file as unreadable and leave the orphan blob probe-first exists to prevent. The NULL is the record. - Frames get previews too. The no-re-probe rule is about metadata an operator reads back as fact; a cache is reported to nobody. - One edge, one cache: `max_edge` is not a parameter on either path. - `_store` fills a NULL on a deduplicated asset and never replaces a value, so re-ingesting a source catches up assets written before the cache existed. Origin still records the first sighting and is never rewritten. `IngestService.backfill_thumbnails(project_id) -> ThumbnailBackfill` is the remedy for the three things a NULL can mean. It reads the blob rather than `asset.uri`, runs in three phases with the rendering in no transaction, and re-reads each asset before writing. It reports rather than raises, on `ReleaseService.verify`'s terms: `missing` (the blob is gone) stays apart from `unreadable` (the bytes will not decode), and only the second reuses `IngestFailure` — `IngestFailureKind` answers "what is wrong with this file", and a blob that is not there is not a file. Migration 10 (`asset_thumbnail`), so `FORMAT_VERSION` is now 10. One nullable column, no foreign key, no data pre-check. Unlike migration 9 it needs its own undo in `_downgrade_to_version_one`: `asset` is only ever altered, so nothing later rebuilds it. The flip side is that the walk back to generation 1 runs its real ALTER, which is why it needs no generation twin of `test_migration_nine_alters_a_table_migration_eight_rebuilt`. 877 tests, up from 854. `VERSION` unchanged, no `openapi.json` drift, no new dependency, no new error class, no new event.
JArmandoAnaya
added a commit
that referenced
this pull request
Aug 21, 2026
…redates them (#21) (#93) Every ingested item now gets a preview, stored content-addressed beside its content and named by `Asset.thumbnail_hash`. The M5 gallery is the reason: decoding full-resolution images per tile at request time is the wrong shape, and the cost amortizes here, where the bytes are already open. A thumbnail hash is a cache key, not an identity — absent from every release manifest, never recomputed by `ReleaseService.verify`, and legitimately different across Pillow builds. Everything else follows from that: - A preview that will not render is NOT an `IngestFailure`. The asset exists and nothing was lost, so the hash stays NULL and the run carries on with an empty report. The `try` is nested inside the caller's, because reaching the outer `except MediaError` would report a good file as unreadable and leave the orphan blob probe-first exists to prevent. The NULL is the record. - Frames get previews too. The no-re-probe rule is about metadata an operator reads back as fact; a cache is reported to nobody. - One edge, one cache: `max_edge` is not a parameter on either path. - `_store` fills a NULL on a deduplicated asset and never replaces a value, so re-ingesting a source catches up assets written before the cache existed. Origin still records the first sighting and is never rewritten. `IngestService.backfill_thumbnails(project_id) -> ThumbnailBackfill` is the remedy for the three things a NULL can mean. It reads the blob rather than `asset.uri`, runs in three phases with the rendering in no transaction, and re-reads each asset before writing. It reports rather than raises, on `ReleaseService.verify`'s terms: `missing` (the blob is gone) stays apart from `unreadable` (the bytes will not decode), and only the second reuses `IngestFailure` — `IngestFailureKind` answers "what is wrong with this file", and a blob that is not there is not a file. Migration 10 (`asset_thumbnail`), so `FORMAT_VERSION` is now 10. One nullable column, no foreign key, no data pre-check. Unlike migration 9 it needs its own undo in `_downgrade_to_version_one`: `asset` is only ever altered, so nothing later rebuilds it. The flip side is that the walk back to generation 1 runs its real ALTER, which is why it needs no generation twin of `test_migration_nine_alters_a_table_migration_eight_rebuilt`. 877 tests, up from 854. `VERSION` unchanged, no `openapi.json` drift, no new dependency, no new error class, no new event.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #21. M2 goes to 7/8; only #23 (close-out and the
v0.0.1-alpha.2tag) is left.Every ingested item now gets a preview, stored content-addressed beside its content and named by
Asset.thumbnail_hash. The M5 gallery is the reason: decoding full-resolution images per tile at request time is the wrong shape, and the cost amortizes at ingest, where the bytes are already open and already decoded once.The constraint everything falls out of
Set by #16, restated in
docs/media.md. It never enters a release manifest,ReleaseService.verifynever recomputes it, and two machines may legitimately hold different preview bytes for one image. Losing every thumbnail blob loses only the time to render them again.Decisions, and why
IngestFailure. That error means "this file did not become an asset, so fix the file"; here the asset exists and nothing was lost. Thetryis nested inside the caller's — letting a refusal reach_read_directory'sexcept MediaErrorwould report a perfectly good file as unreadable and leave the orphan blob that probing first exists to prevent. Nothing is logged: the NULL is the record, and it is exactly what the backfill queries for._read_video's "frames are never re-probed" rule is about metadata an operator reads back as fact. A cache artifact is reported to nobody, and a gallery with tiles for stills and blanks for frames would be the worse outcome.max_edgeis not a parameter on ingest or on the backfill — a per-call edge would fork the cache into variants nothing can tell apart from a hash._storefills a NULL and never replaces a value. Origin fields record the first sighting and are never rewritten; a preview is not provenance, so filling an empty one is not a rewrite. That is what makes re-ingesting a source enough to catch up assets written before the cache existed.asset.uri— that path may be gone, renamed, or on another machine.ReleaseService.verify's terms.missing(the content blob is gone — damage a preview pass cannot repair) stays apart fromunreadable(the bytes are there and will not decode). Only the second reusesIngestFailure:IngestFailureKindanswers "what is wrong with this file", and a blob that is not there is not a file. Raising would abandon the repair of every healthy asset over one bad row.Migration 10 (
asset_thumbnail) —FORMAT_VERSION9 → 10One nullable column. No foreign key, so migration 8's "a column carrying a key cannot arrive by
ALTERat all" limit does not bite. No data pre-check: the column is a cache, so NULL is the ordinary state rather than a legacy value to tolerate.Unlike migration 9 it needs its own undo in
_downgrade_to_version_one. Migration 9's columns rode back on migration 8's rebuild ofingest_job;assetis only ever altered — four cascading children, legitimate pre-pipeline rows — so nothing later removesthumbnail_hashon the way to generation 1. The flip side is that the walk back does run migration 10's realALTER, which is why it needs no generation twin oftest_migration_nine_alters_a_table_migration_eight_rebuilt. Both halves of that reasoning are written down where a future reader will look.Checks
VERSIONstays0.0.1.dev0. No new dependency, no new error class, no new event, no frontend change.Tests
# --- a preview per asset ---— retrievable by hash on both paths, JPEG whatever the source was, bounded by the port constant (never a copied number), never enlarged, repeatable across projects (never a hardcoded hash), a refusal still leaves no blob of either kind, and athumbnailthat raises leaves a NULL with an empty report.# --- the backfill ---— fills what is missing, idempotent on a second pass, reads the blob after the source file is deleted, reports a deleted blob asmissingwhile still filling its neighbour, reports undecodable bytes asunreadablewith the rightkind, and refuses an unknown or foreign project.Fixture.content_blob_count()is new: every dedup claim in that file is about content, and a bareblob_countwould make "one image, one blob" read as two.blob_countkeeps its meaning and is asserted directly where the preview blobs are the subject.