Retire the opt-in strict_provenance runtime guardrail; enforce the make() contract by deploy-time code inspection

[dimitri-yatsenko]

Summary

Remove dj.config["strict_provenance"] (introduced in 2.3.0, #1425) in an upcoming 2.3.x release, along with its documentation. Keep Diagram.trace() (#1423) and self.upstream (#1424). Enforcement of the make() contract moves to static/agentic inspection of make() bodies at deployment (platform CI), backed by governed, commit-pinned execution. The open-source documentation will document the reproducible make() contract itself — the rules — rather than a runtime enforcement flag.

What shipped in 2.3.0

strict_provenance is an opt-in, off-by-default, best-effort development guardrail. It wires read/write checks into the Python client via a per-call contextvars enforcement context, confining reads to self.upstream[...] and writes to self and Parts. It shipped alongside Diagram.trace() and self.upstream as the "provenance trinity."

Why remove it

The shipped specification already concedes the core point: comprehensive enforcement of the make() contract is fundamentally a code-inspection problem, not a runtime-interception one. Against the flag's modest dev-only benefit, the costs are real:

1. Best-effort by construction. The spec's own limits section enumerates the bypasses — existence/count idioms (len, bool, in), restriction-by-table, and any access outside the Python client. It also cannot see code paths a given run does not execute. A guardrail that misses these invites reliance it cannot support.
2. Wrong instrument, wrong time. The make() contract is a static property of the method body (reads confined to the upstream trace cone; writes to self/Parts; one entity per call in isolation; no result-affecting side inputs). Static analysis sees all paths, catches the documented bypass idioms, and reports once in CI before deploy — not per-invocation in the populate hot path.
3. Complexity and maintenance burden. Per-call contextvars context, client-path interception, and a documented-limits appendix are a standing support cost for a check that cannot be made sound in-process.
4. Redundant with the authoritative enforcer. The platform inspects make() source at deploy and controls execution (commit-pinned, mediated connections). That is where a sound guarantee lives.

Why removal (not deprecation) is safe now

• The flag is off by default and has no schema, data, or query-result impact — removal changes no pipeline's behavior for anyone who has not opted in.
• It was introduced in 2.3.0 and has no known adoption: no platform customer has enabled it. Removing it now, before it gains use, is cheaper for everyone than carrying a deprecation cycle for a feature nobody depends on.

What we keep (and stabilize)

Diagram.trace() and self.upstream — the provenance query API and the idiomatic read pattern. "Read from self.upstream, write to self and Parts" remains the documented convention, and it is precisely what makes a make() method statically verifiable.

Proposed changes

• Remove the strict_provenance config key and the runtime interception machinery (enforcement context, client-path checks).
• Remove all mentions from the documentation (2.3 release notes, provenance specification §3); reframe the "trinity" as the trace + self.upstream pair.
• Replace the removed enforcement docs with documentation of the reproducible make() contract: populate-only insertion; one entity per call, computed in isolation; reads confined to the upstream trace cone; writes only to self and Parts; no other result-affecting inputs (populate kwargs orchestrate, never parameterize).
• Optionally, publish the static make() verifier as a standalone dev tool / pre-commit hook, preserving the "catch accidental drift locally" purpose with strictly better coverage than the runtime gate.

Alternatives considered

• Harden the runtime gate — rejected: closing the documented bypasses approaches whole-program analysis inside a running process.
• Deprecate first, remove at 2.4 — unnecessary ceremony given off-by-default, zero functional impact, and no known adoption; removing before the feature gains use is the cheapest path for users and maintainers.

Refs: #1425, #1423, #1424, #1474, #1480, #1485. Docs affected: about/whats-new-23.md, reference/specs/provenance.md.