fix(pipeline): fail fast when the provenance store cannot persist records#662
Merged
Conversation
…ords - Add an optional check() to ProvenanceStore; the pipeline calls it once at the start of a run, so a store that would fail every write fails the run immediately instead of silently disabling skip-unchanged - FileProvenanceStore.check() probes the file's directory with a real write, catching a mount the runtime user cannot write to - Report a record write that fails mid-run through the reporter (stageFailed, stage 'provenance') instead of swallowing it - search-indexer image: pre-create /provenance owned by the runtime user, so a fresh named volume mounted there is writable by uid 1000
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #661
The search-indexer image runs as
node(uid 1000) but wrotePROVENANCE_FILEwithout ensuring its directory is writable by that user. On a root-owned mount – the default for a fresh named volume – every provenance write failed silently (the pipeline swallowed write errors), so skip-unchanged never engaged and every run reprocessed the full dataset.Changes
Fail fast at run start (
@lde/pipeline)ProvenanceStoregains an optionalcheck(); the pipeline calls it once at the start of a run, before selecting datasets. A store that would fail every write now fails the run immediately with a clear error instead of silently disabling skip-unchanged.FileProvenanceStore.check()probes the file’s directory with a real write (create + remove a probe file), catching exactly the root-owned-volume case:Surface mid-run write failures (
@lde/pipeline)stageFailed, stage'provenance') instead of being swallowed.Make the natural configuration just work (search-indexer image)
/provenanceowned bynode, like/data. A fresh named volume mounted there inherits that ownership, soPROVENANCE_FILE=/provenance/provenance.jsonwith-v provenance:/provenanceworks out of the box. Bind mounts still need a host-sidechown, which the boot error now points out.READMEs of both packages document the writability requirement.
Verification
Beyond the unit tests, verified against the built image:
ls -ld /provenance→drwxr-xr-x node node; with a fresh named volume mounted at/provenance, uid 1000 can createprovenance.json(the issue’s failing scenario).PROVENANCE_FILEon a root-owned mount now exits 1 with the error above instead of completing “successfully” without a provenance file.