feat(kernel): SourceService — source registration with provenance (#18) - #90
Merged
Conversation
Replaces the M1 `Source` placeholder with a real record of where a project's raw data came from, and adds `SourceService` as the one door to it. - `domain/source.py`: `SourceKind` (image_directory | video), frozen `VideoProvenance` wrapping the port's `VideoMetadata` plus the chosen `extraction_fps`, and a `Source` carrying `path`, tz-aware `registered_at` and opaque `capture_params`. `uri` is renamed to `path` — it holds `canonical_path()`, an absolute resolved local path. - `SourceService.register_images` / `register_video`: two methods because a clip needs a rate and a probe and a directory needs neither. The probe runs before the transaction opens. Registration is idempotent on `(kind, path, extraction_fps)`; differing capture params or a replaced clip refresh the matched source in place rather than forking it. - Migration 7 rebuilds `source` rather than altering it: `registered_at` is NOT NULL with no honest default, and a pre-#18 row's `kind='local_folder'` is not a value `SourceKind` has. It counts `ingest_job` as well as `source`, because `DROP TABLE` under `PRAGMA foreign_keys = ON` cascades silently. FORMAT_VERSION 6 -> 7. - `SOURCES` becomes a hand-written mapper pair (timestamp + nested JSON). - New `docs/sources.md`; persistence, media, examples docs brought current. No new domain event, no VERSION bump, no openapi drift. Closes #18
2 tasks
JArmandoAnaya
added a commit
that referenced
this pull request
Aug 21, 2026
… (#90) Replaces the M1 `Source` placeholder with a real record of where a project's raw data came from, and adds `SourceService` as the one door to it. - `domain/source.py`: `SourceKind` (image_directory | video), frozen `VideoProvenance` wrapping the port's `VideoMetadata` plus the chosen `extraction_fps`, and a `Source` carrying `path`, tz-aware `registered_at` and opaque `capture_params`. `uri` is renamed to `path` — it holds `canonical_path()`, an absolute resolved local path. - `SourceService.register_images` / `register_video`: two methods because a clip needs a rate and a probe and a directory needs neither. The probe runs before the transaction opens. Registration is idempotent on `(kind, path, extraction_fps)`; differing capture params or a replaced clip refresh the matched source in place rather than forking it. - Migration 7 rebuilds `source` rather than altering it: `registered_at` is NOT NULL with no honest default, and a pre-#18 row's `kind='local_folder'` is not a value `SourceKind` has. It counts `ingest_job` as well as `source`, because `DROP TABLE` under `PRAGMA foreign_keys = ON` cascades silently. FORMAT_VERSION 6 -> 7. - `SOURCES` becomes a hand-written mapper pair (timestamp + nested JSON). - New `docs/sources.md`; persistence, media, examples docs brought current. No new domain event, no VERSION bump, no openapi drift. Closes #18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18. M2 task 4 of 8 (
#22 → #16 → #17 → **#18** → #20 → #19 → #21 → #23).What lands
SourceServiceis the one door to aSource— the record that raw data was offered to a project. It holds no pixels; assets are what an ingest materializes from it.Two registration methods rather than one
register(kind=...): a clip needs a rate and gets probed, a directory needs neither and is not walked — the same argument that madeImageProcessorandVideoProcessortwo protocols.Decisions worth reviewing
uriis renamed topath. The field holdscanonical_path()—str(Path.resolve(strict=True)), absolute with symlinks followed. Migration 7 is a rebuild, so the rename is free now; after #20 joins onasset.source_idit would not be.Decomposition params live on the source, not the ingest job. "Same source, same assets" only means something if the parameters are part of what the same source is. Consequence, deliberate: one clip at 1 fps and again at 5 fps is two sources.
Registration is idempotent on
(kind, path, extraction_fps). Excludingcapture_params(a typo in a lens note must not fork one directory into two origins) and excluding the probedVideoMetadata(a clip replaced at a known path is still that path's source — its provenance is refreshed in place, so no record survives describing bytes nobody can produce).registered_atis never rewritten.The idempotency has no unique index under it, and that gap is named, not hidden. Documented in the service module docstring and
docs/sources.md: a duplicate source is inert today because nothing references one, and #20 — whereasset.source_idgets a target — is where this needs the index.Migration 7 rebuilds
sourcerather than altering it (FORMAT_VERSION6 → 7), on migration 6's terms:registered_atis NOT NULL with no honest default, and a pre-#18 row'skind='local_folder'is not a valueSourceKindhas, so those rows would come back as validation errors. This supersedes the roadmap note that assumed anALTER— with a rebuild the ALTER-adds-last rule has nothing to bite on.Migration 7 counts
ingest_jobas well assource.ingest_job.source_idisON DELETE CASCADEand the store setsPRAGMA foreign_keys = ONon every connection, soDROP TABLE sourceruns an implicitDELETEthat takes the jobs with it silently, without raising. Migration 6 dropped a table with no children and so never met this; there is a parametrized test for both halves.Sourceis the only domain model withvalidate_assignment=True. Thevideo is not None ⟺ kind is VIDEOrule is amodel_validator(mode="after"), which does not re-run on assignment; without itsource.kind = IMAGE_DIRECTORYwould leave a populatedvideobehind.require_video()reads it, so mypy never sees aVideoProvenance | Nonethat callers must assert away.SourceKindis an enum whereDatasetChange.operationis astr. That doctrine turns on can something outside this build write the value? — no foreign writer exists, the kernel branches on it, and the set grows deliberately. The argument is written into the enum's docstring so it does not read as an oversight.Also worth knowing
register_videoprobes before opening the transaction — an out-of-process decoder inside a write transaction is how a single-writer SQLite store starts reporting "database is locked". The corollary is documented: re-registering a known clip still needs ffmpeg.Registration is not a validation pass. A truncated faststart clip has a readable header, so it registers fine and records the intact duration; damage surfaces at extraction. There is a test asserting exactly that, so #19/#20 do not assume otherwise.
Not in this PR
No domain event (
IngestCompletedis #20's tripwire to flip). NoSourceService.delete. NoVERSIONbump, noopenapi.jsondrift, no dependency change.Checks