Highlights
DataJoint 2.3 builds on the 2.2.x line with a set of features centered on provenance — knowing, and being able to trust, exactly which upstream data a computed result was derived from.
The headline is the provenance trinity. Diagram.trace constructs the upstream view of any result; self.upstream makes those declared ancestors ergonomic to read inside make(); and the opt-in strict_provenance flag turns the long-standing "read only from declared dependencies, write only to self" convention into something the framework actively checks. Together they move DataJoint's core provenance promise from a convention people hope to follow to one the framework helps construct and — when enabled — enforce.
Around that:
- SparkAdapter Codec Protocol — typed codecs can expose their decoded values as Spark-native types, opening lakehouse / Delta-Sharing consumers (e.g. Databricks) to columns that were previously opaque blobs.
dj.deploy.set_replica_identity— a newdj.deploymodule configures PostgreSQLREPLICA IDENTITYfor change-data-capture pipelines.- Cascade fix —
part_integrity="cascade"is now correct across Part-of-Part and renamed-foreign-key chains (the same upward-propagation machineryDiagram.tracebuilds on).
No breaking changes. strict_provenance defaults off and everything else is additive, so existing pipelines are unaffected. See What's New in 2.3 for the full narrative.
🚀 Features
- feat(#1425): strict_provenance config flag for runtime enforcement(#1474)@dimitri-yatsenko
- feat(#1424): self.upstream property for pre-restricted ancestor access(#1473)@dimitri-yatsenko
- feat(#1423): Diagram.trace() for upstream restriction propagation(#1471)@dimitri-yatsenko
- feat(#1458): SparkAdapter Codec Protocol(#1472)@dimitri-yatsenko
- feat(deploy): set_replica_identity for PostgreSQL CDC (#1447)(#1466)@dimitri-yatsenko
🐛 Bug Fixes
- fix(#1429): cascade through FK chain for part_integrity="cascade"(#1468)@dimitri-yatsenko
- fix(#1454): refresh ~lineage on every
@schemadecoration(#1467)@dimitri-yatsenko - fix(staged_insert): converge metadata shape with ObjectCodec.encode(#1465)@dimitri-yatsenko
Full Changelog: v2.2.4...v2.3.0
