DataJoint Library 2.3.4

A patch release on the 2.3 line, focused on where data came from before it reached the pipeline.

Rows that enter the pipeline can now record where they came from. With provenance.capture enabled, a dj.Manual table carries a hidden _prov attribute, declared with the table and filled on insert. It is off by default, like jobs.add_job_metadata and for the same reason: it changes the DDL of every entry table declared afterwards, so that is a deployment's decision rather than a library default. Inside a pipeline provenance is structural — a computed row cannot exist without its declared upstream, so the foreign-key graph is the lineage — but at the boundary that runs out, and until now each pipeline invented its own record of external origin. _prov gives the boundary one shape, so that "which rows have no recorded origin" is a query rather than an audit. No author writes it: insert() takes no provenance argument, and the content comes from deployment configuration (provenance.source), from the connection, and — for an insert inside a make() — from the ingesting table and key, which is what makes the fan-out ingestion pattern traceable without a foreign key. A table declared before capture was enabled records nothing until dj.deploy.add_prov_column adds the slot. See Extrinsic Provenance at Entry Tables and Record Data Origin.

Three tiers gain a second name. dj.Entry, dj.Ingest and dj.Compute join dj.Manual, dj.Imported and dj.Computed. Each pair is one class rather than a subclass, so a table declared either way is identical — same SQL prefix, same tier detection, same describe() and diagram — and both names are permanent, with no deprecation and nothing to migrate. dj.Lookup and dj.Part are unchanged. The documentation keeps the original names for now; adopting the new ones as primary is slated for 2.4 (#1546).

Codecs receive connection context explicitly. encode() and decode() now take a context argument carrying schema, table, field and config, so key means what it means everywhere else — the primary key. Passing the calling connection's config is what matters: without it, path and backend resolution fall back to the global dj.config, which in a process holding connections for several users belongs to none of them and resolves a different store silently. Codecs written before 2.3.4 keep working unchanged; DataJoint passes context only to codecs whose signature declares it.

Two PostgreSQL defects in the hidden columns are fixed. Every platform-managed column — _job_start_time, _job_duration, _job_version, _prov, _singleton — is now declared in DataJoint notation and compiled through the type system, so each backend's mapping comes from one place. _job_start_time had been declared as a bare timestamp and so took microsecond precision on PostgreSQL where MySQL got datetime(3), and migrate.add_job_metadata_columns built its ALTER with backtick-quoted identifiers, which is a syntax error there — the retrofit had never run on PostgreSQL at all. Existing tables are untouched; only new declarations and retrofits change.

Upgrading from 2.3.3? No API breaks, and no DDL change: every new setting is off by default, so an unchanged schema declares under 2.3.4 exactly what it declared under 2.3.3. Reading a hidden attribute still requires SQL — to_arrays('_prov') and proj('_prov') raise, and a supported accessor arrives in 2.4 (#1562).

Full details in What's New in DataJoint 2.3.

---

What's Changed

• Pull MinIO from a registry that still serves it(#1559)@dimitri-yatsenko

⚡️ Enhancements

• Provenance capture defaults to off(#1568)@dimitri-yatsenko
• Codec context argument, additive to key(#1556)@dimitri-yatsenko
• Hidden _prov attribute for extrinsic provenance at Entry tables(#1555)@dimitri-yatsenko
• dj.Entry, dj.Ingest and dj.Compute as tier names on one axis(#1558)@dimitri-yatsenko

🐛 Bug Fixes

• Declare every platform-managed column through the DataJoint type system(#1565)@dimitri-yatsenko

Full Changelog: v2.3.3...v2.3.4