Skip to content

feat(kernel): SourceService — source registration with provenance (#18) - #90

Merged
JArmandoAnaya merged 1 commit into
mainfrom
feat/source-service
Jul 27, 2026
Merged

feat(kernel): SourceService — source registration with provenance (#18)#90
JArmandoAnaya merged 1 commit into
mainfrom
feat/source-service

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Closes #18. M2 task 4 of 8 (#22 → #16 → #17 → **#18** → #20 → #19 → #21 → #23).

What lands

SourceService is the one door to a Source — the record that raw data was offered to a project. It holds no pixels; assets are what an ingest materializes from it.

sources = SourceService(workspace)
stills = sources.register_images(project.id, Path("captures/2026-07"))
clip   = sources.register_video(project.id, Path("drive.mp4"), extraction_fps=5.0)
clip.require_video().metadata.fps        # the ORIGINAL rate, from VideoProcessor.probe
clip.require_video().extraction_fps      # the rate a decomposition will run at

Two registration methods rather than one register(kind=...): a clip needs a rate and gets probed, a directory needs neither and is not walked — the same argument that made ImageProcessor and VideoProcessor two protocols.

Decisions worth reviewing

uri is renamed to path. The field holds canonical_path()str(Path.resolve(strict=True)), absolute with symlinks followed. Migration 7 is a rebuild, so the rename is free now; after #20 joins on asset.source_id it would not be.

Decomposition params live on the source, not the ingest job. "Same source, same assets" only means something if the parameters are part of what the same source is. Consequence, deliberate: one clip at 1 fps and again at 5 fps is two sources.

Registration is idempotent on (kind, path, extraction_fps). Excluding capture_params (a typo in a lens note must not fork one directory into two origins) and excluding the probed VideoMetadata (a clip replaced at a known path is still that path's source — its provenance is refreshed in place, so no record survives describing bytes nobody can produce). registered_at is never rewritten.

The idempotency has no unique index under it, and that gap is named, not hidden. Documented in the service module docstring and docs/sources.md: a duplicate source is inert today because nothing references one, and #20 — where asset.source_id gets a target — is where this needs the index.

Migration 7 rebuilds source rather than altering it (FORMAT_VERSION 6 → 7), on migration 6's terms: registered_at is NOT NULL with no honest default, and a pre-#18 row's kind='local_folder' is not a value SourceKind has, so those rows would come back as validation errors. This supersedes the roadmap note that assumed an ALTER — with a rebuild the ALTER-adds-last rule has nothing to bite on.

Migration 7 counts ingest_job as well as source. ingest_job.source_id is ON DELETE CASCADE and the store sets PRAGMA foreign_keys = ON on every connection, so DROP TABLE source runs an implicit DELETE that takes the jobs with it silently, without raising. Migration 6 dropped a table with no children and so never met this; there is a parametrized test for both halves.

Source is the only domain model with validate_assignment=True. The video is not None ⟺ kind is VIDEO rule is a model_validator(mode="after"), which does not re-run on assignment; without it source.kind = IMAGE_DIRECTORY would leave a populated video behind. require_video() reads it, so mypy never sees a VideoProvenance | None that callers must assert away.

SourceKind is an enum where DatasetChange.operation is a str. That doctrine turns on can something outside this build write the value? — no foreign writer exists, the kernel branches on it, and the set grows deliberately. The argument is written into the enum's docstring so it does not read as an oversight.

Also worth knowing

register_video probes before opening the transaction — an out-of-process decoder inside a write transaction is how a single-writer SQLite store starts reporting "database is locked". The corollary is documented: re-registering a known clip still needs ffmpeg.

Registration is not a validation pass. A truncated faststart clip has a readable header, so it registers fine and records the intact duration; damage surfaces at extraction. There is a test asserting exactly that, so #19/#20 do not assume otherwise.

Not in this PR

No domain event (IngestCompleted is #20's tripwire to flip). No SourceService.delete. No VERSION bump, no openapi.json drift, no dependency change.

Checks

uv run ruff format .          110 files left unchanged
uv run ruff check .           All checks passed!
uv run mypy src/visionset/kernel   Success: no issues found in 49 source files
uv run lint-imports           Contracts: 2 kept, 0 broken.
VISIONSET_REQUIRE_FFMPEG=1 uv run pytest   769 passed
uv run python scripts/export_openapi.py    no diff

Replaces the M1 `Source` placeholder with a real record of where a project's
raw data came from, and adds `SourceService` as the one door to it.

- `domain/source.py`: `SourceKind` (image_directory | video), frozen
  `VideoProvenance` wrapping the port's `VideoMetadata` plus the chosen
  `extraction_fps`, and a `Source` carrying `path`, tz-aware `registered_at`
  and opaque `capture_params`. `uri` is renamed to `path` — it holds
  `canonical_path()`, an absolute resolved local path.
- `SourceService.register_images` / `register_video`: two methods because a
  clip needs a rate and a probe and a directory needs neither. The probe runs
  before the transaction opens. Registration is idempotent on
  `(kind, path, extraction_fps)`; differing capture params or a replaced clip
  refresh the matched source in place rather than forking it.
- Migration 7 rebuilds `source` rather than altering it: `registered_at` is
  NOT NULL with no honest default, and a pre-#18 row's `kind='local_folder'`
  is not a value `SourceKind` has. It counts `ingest_job` as well as `source`,
  because `DROP TABLE` under `PRAGMA foreign_keys = ON` cascades silently.
  FORMAT_VERSION 6 -> 7.
- `SOURCES` becomes a hand-written mapper pair (timestamp + nested JSON).
- New `docs/sources.md`; persistence, media, examples docs brought current.

No new domain event, no VERSION bump, no openapi drift.

Closes #18
@JArmandoAnaya
JArmandoAnaya merged commit ca97598 into main Jul 27, 2026
3 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/source-service branch July 27, 2026 11:02
JArmandoAnaya added a commit that referenced this pull request Aug 21, 2026
… (#90)

Replaces the M1 `Source` placeholder with a real record of where a project's
raw data came from, and adds `SourceService` as the one door to it.

- `domain/source.py`: `SourceKind` (image_directory | video), frozen
  `VideoProvenance` wrapping the port's `VideoMetadata` plus the chosen
  `extraction_fps`, and a `Source` carrying `path`, tz-aware `registered_at`
  and opaque `capture_params`. `uri` is renamed to `path` — it holds
  `canonical_path()`, an absolute resolved local path.
- `SourceService.register_images` / `register_video`: two methods because a
  clip needs a rate and a probe and a directory needs neither. The probe runs
  before the transaction opens. Registration is idempotent on
  `(kind, path, extraction_fps)`; differing capture params or a replaced clip
  refresh the matched source in place rather than forking it.
- Migration 7 rebuilds `source` rather than altering it: `registered_at` is
  NOT NULL with no honest default, and a pre-#18 row's `kind='local_folder'`
  is not a value `SourceKind` has. It counts `ingest_job` as well as `source`,
  because `DROP TABLE` under `PRAGMA foreign_keys = ON` cascades silently.
  FORMAT_VERSION 6 -> 7.
- `SOURCES` becomes a hand-written mapper pair (timestamp + nested JSON).
- New `docs/sources.md`; persistence, media, examples docs brought current.

No new domain event, no VERSION bump, no openapi drift.

Closes #18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernel: SourceService — source registration with provenance metadata (origin, date, original framerate, parameters)

1 participant