Retire strict_provenance runtime guardrail (#1488)#1489
Open
dimitri-yatsenko wants to merge 2 commits into
Open
Retire strict_provenance runtime guardrail (#1488)#1489dimitri-yatsenko wants to merge 2 commits into
dimitri-yatsenko wants to merge 2 commits into
Conversation
Remove the opt-in `dj.config["strict_provenance"]` flag (introduced in 2.3.0, #1425) and its runtime interception machinery. Enforcement of the make() reproducibility contract moves to deploy-time static/agentic inspection of make() bodies on the platform; the open-source docs document the contract itself rather than a runtime flag. Kept intact: `Diagram.trace()` (#1423) and `self.upstream` (#1424) — the upstream query API and the idiomatic "read from self.upstream, write to self" pattern that makes a make() statically verifiable. Changes: - Delete src/datajoint/provenance.py (enforcement context + read/write/ key-consistency gates — the whole module supported only the flag). - settings.py: drop the strict_provenance config field and its DJ_STRICT_PROVENANCE env-var mapping. - expression.py: remove the read gate from QueryExpression.cursor. - table.py: remove the write gate and per-row key-consistency check from the insert path. - autopopulate.py: remove the per-make() strict context push/pop; keep the self._upstream = Diagram.trace(self & key) construction and the `upstream` property (docstring reworded to drop the flag reference). - Delete tests/integration/test_strict_provenance.py. Removal (not deprecation) is safe: the flag is off by default, has no schema/data/query-result impact, and has no known adoption.
DataJoint's own docs use the relational workflow model's vocabulary (lineage, reproducibility); provenance is a platform-level compatibility concern documented separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1488.
Summary
Removes the opt-in
dj.config["strict_provenance"]flag (introduced in 2.3.0, #1425) and its runtime interception machinery. Comprehensive enforcement of themake()contract is fundamentally a code-inspection problem, not a runtime-interception one — the shipped guardrail was best-effort by construction (documented bypasses via existence/count idioms, restriction-by-table, and any access outside the Python client) and could not be made sound in-process. Enforcement moves to deploy-time static/agentic inspection ofmake()bodies on the platform, backed by governed, commit-pinned execution. The open-source docs document the make() reproducibility contract itself (the rules) rather than a runtime flag.Kept (and stabilized)
Diagram.trace()(Implement Diagram.trace() for upstream restriction propagation #1423)self.upstream(Add self.upstream property for pre-restricted ancestor access in make() #1424)"Read from
self.upstream, write toselfand Parts" remains the documented convention — and it is exactly what makes amake()statically verifiable. These are cleanly separable from the removed enforcement code:self._upstream = Diagram.trace(self & key)still runs unconditionally.Changes
src/datajoint/provenance.py— the entire module (enforcement context, read/write gates, per-row key-consistency check) existed only to back the flag.settings.py— drop thestrict_provenanceconfig field and itsDJ_STRICT_PROVENANCEenv-var mapping.expression.py— remove the read gate fromQueryExpression.cursor.table.py— remove the write gate and the per-row key-consistency check from the insert path.autopopulate.py— remove the per-make()strict context push/pop; keep the upstream-trace construction and theupstreamproperty (docstring reworded to drop the flag reference and point at the AutoPopulate spec).tests/integration/test_strict_provenance.py.Why removal, not deprecation
The flag is off by default, has no schema/data/query-result impact, and has no known adoption (no platform customer enabled it). Removing it before it gains use is cheaper than carrying a deprecation cycle for a feature nobody depends on.
Docs
The
strict_provenancedocumentation (2.3 release notes, provenance spec §3) lives in the separatedatajoint-docsrepo and is handled there — the reproduciblemake()contract is being documented inreference/specs/autopopulate.md(datajoint-docs #197), and the "trinity" reframed to thetrace+self.upstreampair.Verification
datajointimports cleanly (2.3.0);strict_provenancekey removed;Diagram.traceandself.upstreamintact;datajoint.provenanceno longer importable.src//tests/.ruff,ruff-format, andcodespellpre-commit hooks pass. (Full integration test run requires a live database, not available in the authoring environment.)