Skip to content

Retire strict_provenance runtime guardrail (#1488)#1489

Open
dimitri-yatsenko wants to merge 2 commits into
masterfrom
retire/strict-provenance
Open

Retire strict_provenance runtime guardrail (#1488)#1489
dimitri-yatsenko wants to merge 2 commits into
masterfrom
retire/strict-provenance

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Closes #1488.

Summary

Removes the opt-in dj.config["strict_provenance"] flag (introduced in 2.3.0, #1425) and its runtime interception machinery. Comprehensive enforcement of the make() contract is fundamentally a code-inspection problem, not a runtime-interception one — the shipped guardrail was best-effort by construction (documented bypasses via existence/count idioms, restriction-by-table, and any access outside the Python client) and could not be made sound in-process. Enforcement moves to deploy-time static/agentic inspection of make() bodies on the platform, backed by governed, commit-pinned execution. The open-source docs document the make() reproducibility contract itself (the rules) rather than a runtime flag.

Kept (and stabilized)

"Read from self.upstream, write to self and Parts" remains the documented convention — and it is exactly what makes a make() statically verifiable. These are cleanly separable from the removed enforcement code: self._upstream = Diagram.trace(self & key) still runs unconditionally.

Changes

  • Delete src/datajoint/provenance.py — the entire module (enforcement context, read/write gates, per-row key-consistency check) existed only to back the flag.
  • settings.py — drop the strict_provenance config field and its DJ_STRICT_PROVENANCE env-var mapping.
  • expression.py — remove the read gate from QueryExpression.cursor.
  • table.py — remove the write gate and the per-row key-consistency check from the insert path.
  • autopopulate.py — remove the per-make() strict context push/pop; keep the upstream-trace construction and the upstream property (docstring reworded to drop the flag reference and point at the AutoPopulate spec).
  • Delete tests/integration/test_strict_provenance.py.

Why removal, not deprecation

The flag is off by default, has no schema/data/query-result impact, and has no known adoption (no platform customer enabled it). Removing it before it gains use is cheaper than carrying a deprecation cycle for a feature nobody depends on.

Docs

The strict_provenance documentation (2.3 release notes, provenance spec §3) lives in the separate datajoint-docs repo and is handled there — the reproducible make() contract is being documented in reference/specs/autopopulate.md (datajoint-docs #197), and the "trinity" reframed to the trace + self.upstream pair.

Verification

  • datajoint imports cleanly (2.3.0); strict_provenance key removed; Diagram.trace and self.upstream intact; datajoint.provenance no longer importable.
  • No dangling references to the flag or the removed helpers anywhere in src//tests/.
  • ruff, ruff-format, and codespell pre-commit hooks pass. (Full integration test run requires a live database, not available in the authoring environment.)

Remove the opt-in `dj.config["strict_provenance"]` flag (introduced in
2.3.0, #1425) and its runtime interception machinery. Enforcement of the
make() reproducibility contract moves to deploy-time static/agentic
inspection of make() bodies on the platform; the open-source docs
document the contract itself rather than a runtime flag.

Kept intact: `Diagram.trace()` (#1423) and `self.upstream` (#1424) — the
upstream query API and the idiomatic "read from self.upstream, write to
self" pattern that makes a make() statically verifiable.

Changes:
- Delete src/datajoint/provenance.py (enforcement context + read/write/
  key-consistency gates — the whole module supported only the flag).
- settings.py: drop the strict_provenance config field and its
  DJ_STRICT_PROVENANCE env-var mapping.
- expression.py: remove the read gate from QueryExpression.cursor.
- table.py: remove the write gate and per-row key-consistency check from
  the insert path.
- autopopulate.py: remove the per-make() strict context push/pop; keep
  the self._upstream = Diagram.trace(self & key) construction and the
  `upstream` property (docstring reworded to drop the flag reference).
- Delete tests/integration/test_strict_provenance.py.

Removal (not deprecation) is safe: the flag is off by default, has no
schema/data/query-result impact, and has no known adoption.
DataJoint's own docs use the relational workflow model's vocabulary
(lineage, reproducibility); provenance is a platform-level compatibility
concern documented separately.
@dimitri-yatsenko dimitri-yatsenko added the enhancement Indicates new improvements label Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Indicates new improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retire the opt-in strict_provenance runtime guardrail; enforce the make() contract by deploy-time code inspection

1 participant