Skip to content

feat(examples): ingest end-to-end — a synthetic clip to an approved, partitioned batch (#23) - #94

Merged
JArmandoAnaya merged 1 commit into
mainfrom
feat/ingest-e2e
Jul 27, 2026
Merged

feat(examples): ingest end-to-end — a synthetic clip to an approved, partitioned batch (#23)#94
JArmandoAnaya merged 1 commit into
mainfrom
feat/ingest-e2e

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Closes #23.

M2's exit criterion made executable, the way examples/sdk_end_to_end.py served M1's. A
ten-second testsrc clip and a folder of stills go in; fifty hash-deduplicated assets in an
approved batch of two jobs
come out, with a pollable progress row and a per-file report of what
could not be read. Nothing is annotated and nothing is released — the SDK example already covers
that half of the cycle, and this one spends its length on where assets come from.

What it drives

Stage Result
project + schema v1 one class, because approval needs a version to pin
register_video(..., extraction_fps=5.0)ingest 50 assets in a draft batch, each with a frame index and timestamp
IngestService.get(job_id) state=completed, processed=50, total=None
approve(BySize(size=25)) 2 jobs of 25, schema pinned, 50 awaiting an annotator
re-ingest the same source created=0, deduplicated=50, into a new batch
register_video(..., extraction_fps=1.0) a different source; 10 frames, created=0
3 stills + notes.txt total=4, processed=4, created=3, one IngestFailure
after all four ingests 53 assets, 53 with a thumbnail_hash

Four properties demonstrated rather than described

A clip cannot state its total, and a directory can. processed climbs to 50 while total
stays NULL — VideoMetadata carries no frame count by design, because it would be a guess for a
VFR clip and the number an ingest wants is what extraction produced. The directory can be
listed, so it states 4 before reading its first file.

One file registered at two rates is two sources whose frames are one set. Decomposition
parameters live on the source, so 5.0 and 1.0 over the same path are two origins. And yet
the coarse run creates nothing: identity is content, and round=up lands both grids on whole
seconds, so its ten frames are byte-for-byte frames the finer run already stored. Verified before
writing the assertion — all 10 of 10 coarse frames hash-match, at matching timestamps. The
example and the test both say this is a property of this extractor, not a promise the port
makes about every rate pair.

A file that is not an image is reported, not skipped. One IngestFailure
(UNSUPPORTED, reason not repeating the name so a surface can group by kind) and the run still
ends completed.

Re-ingesting a source creates nothing. Into a new batch, because the first froze at
approval — a batch is an ephemeral unit of work and two of them may name the same assets.

One finding worth recording

The clip is 160×120, not the fixtures' 64×48, and that is load-bearing. testsrc moves a
little between frames; below roughly 96×72 that movement falls under what the scaler and encoder
still resolve, and consecutive frames come out byte-identical. Content addressing then does
exactly what it promises and collapses them — the first working version of this example reported
40 assets from 50 extraction slots, the feature working and reading as a shortfall. Measured
across six sizes: 64×48 gives 40/50 distinct, everything from 96×72 up gives 50/50. The constant
carries the explanation.

Two deliberate divergences from the SDK example

Both stated in the module docstring:

  1. The clip is generated by shelling out to ffmpeg. A video is a container wrapped around a
    codec, and the only honest way to write one is the tool that reads it. The command is
    tests/fixtures/media.write_video's, duplicated rather than imported — that module is a
    test fixture, it imports pytest, and its answer to a missing binary is pytest.skip, which
    means nothing in a script. The example checks shutil.which("ffmpeg") before writing anything
    and exits with an install hint instead.
  2. The stills are Pillow's work, not a second copy of M1's hand-rolled PNG encoder. Pillow
    has been a dependency since kernel: MediaProcessor adapter — Pillow for image validation/dimensions/thumbnails #16. This also retires the now-false "M1 has no image library to
    lean on (Pillow arrives with kernel: MediaProcessor adapter — Pillow for image validation/dimensions/thumbnails #16)" lines in docs/examples.md and examples/README.md.

CI

Runs twice, as the SDK example does: the smoke test calls main() and asserts on the returned
Summary, and a new Ingest end-to-end example step runs the file as a plain script — the only
thing that proves it works from a clean checkout. The python job already installs ffmpeg and
sets VISIONSET_REQUIRE_FFMPEG=1, so the smoke test's require_ffmpeg() gate is a skip locally
and an error on the runner.

Scope

No new service, domain model, error, event or migration. FORMAT_VERSION stays 10, VERSION
stays 0.0.1.dev0, openapi.json unchanged (drift gate run locally, clean). Docs updated:
docs/examples.md (restructured for two examples), docs/README.md, docs/ingest.md,
examples/README.md, root README.md.

Checks

885 passed, 1 warning in 71.99s          (up from 877)
lint-imports                Contracts: 2 kept, 0 broken.
mypy src/visionset/kernel   Success: no issues found in 51 source files
mypy src/visionset          Success: no issues found in 61 source files
ruff check .                All checks passed!
ruff format --check .       116 files already formatted
scripts/export_openapi.py   no drift
uv run python examples/sdk_end_to_end.py        green
uv run python examples/ingest_end_to_end.py     green, and green again on a re-run

…partitioned batch (#23)

M2's exit criterion made executable, the way examples/sdk_end_to_end.py served
M1's. A ten-second testsrc clip and a folder of stills go in; fifty
hash-deduplicated assets in an approved batch of two jobs come out, with a
pollable progress row and a per-file report of what could not be read. Nothing
is annotated and nothing is released — the SDK example already covers that half
of the cycle, and this one spends its length on where assets come from.

Four properties demonstrated rather than described:

- A clip cannot state its total and a directory can. processed climbs to 50
  while total stays NULL, because VideoMetadata carries no frame count by
  design; the image directory states 4 before reading its first file.
- One file registered at two rates is two sources whose frames are one set.
  Decomposition parameters live on the source, so 5 fps and 1 fps over the same
  path are two origins — and the coarse run still creates nothing, because
  identity is content and round=up lands both grids on whole seconds. That
  alignment is a property of this extractor, not a promise the port makes.
- A file that is not an image is reported, not skipped. notes.txt yields one
  IngestFailure and the run still ends completed.
- Re-ingesting a source creates nothing: created=0, deduplicated=50, into a new
  batch because the first froze at approval.

The clip is 160x120 rather than the fixtures' 64x48, and that is load-bearing.
testsrc moves a little between frames; below roughly 96x72 that movement falls
under what the scaler and encoder still resolve, consecutive frames come out
byte-identical, and content addressing collapses them — a ten-second clip at
5 fps then yields forty assets, the feature working and reading as a shortfall.

Two deliberate divergences from the SDK example, both stated in the module
docstring: the clip is generated by shelling out to ffmpeg, because a video is a
container wrapped around a codec and the only honest way to write one is the
tool that reads it; and the stills are Pillow's work rather than a second copy
of M1's hand-rolled PNG encoder, Pillow having been a dependency since #16. The
generation command is duplicated from tests/fixtures/media.py rather than
imported — that module is a test fixture, it imports pytest, and its answer to a
missing binary is pytest.skip, which means nothing in a script. The example
checks shutil.which("ffmpeg") before writing anything and exits with an install
hint instead.

CI runs it twice, as the SDK example is: the smoke test calls main() and asserts
on the returned Summary, and a new "Ingest end-to-end example" step runs the
file as a plain script, which is the only thing that proves it works from a
clean checkout. The smoke test gates on tests/fixtures/media.require_ffmpeg() —
a skip locally, an error under VISIONSET_REQUIRE_FFMPEG=1.

No new service, domain model, error, event or migration. FORMAT_VERSION stays
10, VERSION stays 0.0.1.dev0, openapi.json unchanged.

885 tests, up from 877.
@JArmandoAnaya
JArmandoAnaya merged commit 56fa321 into main Jul 27, 2026
3 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/ingest-e2e branch July 27, 2026 14:47
JArmandoAnaya added a commit that referenced this pull request Aug 21, 2026
…partitioned batch (#23) (#94)

M2's exit criterion made executable, the way examples/sdk_end_to_end.py served
M1's. A ten-second testsrc clip and a folder of stills go in; fifty
hash-deduplicated assets in an approved batch of two jobs come out, with a
pollable progress row and a per-file report of what could not be read. Nothing
is annotated and nothing is released — the SDK example already covers that half
of the cycle, and this one spends its length on where assets come from.

Four properties demonstrated rather than described:

- A clip cannot state its total and a directory can. processed climbs to 50
  while total stays NULL, because VideoMetadata carries no frame count by
  design; the image directory states 4 before reading its first file.
- One file registered at two rates is two sources whose frames are one set.
  Decomposition parameters live on the source, so 5 fps and 1 fps over the same
  path are two origins — and the coarse run still creates nothing, because
  identity is content and round=up lands both grids on whole seconds. That
  alignment is a property of this extractor, not a promise the port makes.
- A file that is not an image is reported, not skipped. notes.txt yields one
  IngestFailure and the run still ends completed.
- Re-ingesting a source creates nothing: created=0, deduplicated=50, into a new
  batch because the first froze at approval.

The clip is 160x120 rather than the fixtures' 64x48, and that is load-bearing.
testsrc moves a little between frames; below roughly 96x72 that movement falls
under what the scaler and encoder still resolve, consecutive frames come out
byte-identical, and content addressing collapses them — a ten-second clip at
5 fps then yields forty assets, the feature working and reading as a shortfall.

Two deliberate divergences from the SDK example, both stated in the module
docstring: the clip is generated by shelling out to ffmpeg, because a video is a
container wrapped around a codec and the only honest way to write one is the
tool that reads it; and the stills are Pillow's work rather than a second copy
of M1's hand-rolled PNG encoder, Pillow having been a dependency since #16. The
generation command is duplicated from tests/fixtures/media.py rather than
imported — that module is a test fixture, it imports pytest, and its answer to a
missing binary is pytest.skip, which means nothing in a script. The example
checks shutil.which("ffmpeg") before writing anything and exits with an install
hint instead.

CI runs it twice, as the SDK example is: the smoke test calls main() and asserts
on the returned Summary, and a new "Ingest end-to-end example" step runs the
file as a plain script, which is the only thing that proves it works from a
clean checkout. The smoke test gates on tests/fixtures/media.require_ffmpeg() —
a skip locally, an error under VISIONSET_REQUIRE_FFMPEG=1.

No new service, domain model, error, event or migration. FORMAT_VERSION stays
10, VERSION stays 0.0.1.dev0, openapi.json unchanged.

885 tests, up from 877.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

release: ingest e2e (synthetic video → 50 frames → approved batch) + 0.0.1-alpha.2 close-out

1 participant