ETLantic 0.34.0

[0.34.0] - 2026-07-30

Security and reliability hardening

• Phase-aware PMPLUG402: authorize/selected-engine denials block
  ensure_plugins / run / compile / doctor / execute (discovery siblings stay
  non-blocking); run maps all raised PMPLUG* to TRUST_FAILURE; doctor
  plugin trust failures exit 11
• Manual overlays cannot spoof builtin local/env allowlist exemption;
  secret / observability / run-history / event-consumer / storage registers are
  production-allowlist gated; authorize path no longer exempts engine-name spoofs
• Profile constructor rejects plaintext secrets; plan/profile serialization
  refuses non-SecretRef values
• Profile assets / binding deserialize reject URL userinfo so credential URLs
  cannot enter plan snapshots; report nested metadata bags and extension keys
  (including dsn / connection_string / jdbc_url and URL values) fail
  closed; CLI emit_payload redacts before echo
• Delta merge_keys validated as safe identifiers; non-Delta MERGE/UPSERT
  fails closed (no overwrite fallback); Delta capabilities require
  delta.tables.DeltaTable
• SQL executor diagnostics redact exception text; PostgreSQL MERGE without
  update columns fails closed unless explicit insert-ignore intent
• Airflow compile preserves schedule timezone (ScheduleIntent.from_mapping
  no longer invents UTC), stubs on any ERROR diagnostic (not only schedule),
  refuses unsupported schedules with a non-schedulable stub, and secret-scans
  rendered source (including URL userinfo)
• Trusted-SQL profile mismatch uses PMEXEC353 and is CONTINUE/SKIP hard-fail;
  redact_message redacts full URL userinfo
• CSV/JSON sinks honor write_mode and optional SafeIoPolicy; memory
  merge/upsert fails closed; FileStateStore.commit reads previous inside
  locked RMW; durable_audit event consumers fail closed
• Medallantic: missing validation/artifact status fail closed; run status
  reconciled when steps failed; empty SparkForge write mode rejected
• FileStateStore fails closed on corrupt state; schema history verifies
  stored fingerprints; file schema/run-history providers default
  fail_closed=True

Added

• Lifecycle event correlation (etlantic.lifecycle_event/1) with plan/region/backend fields
• Observability provider protocol (etlantic.observability/1) and runtime bridge
• Run history provider protocol with file and in-memory reference implementations
• Event consumer protocol with reference trend consumer
• Profile keys: observability_providers, run_history_provider, event_consumers, observability_delivery
• Conformance suites: observability, run history, event consumer, production profile
• Medallantic M6: explain_medallion_plan, lifecycle views, medallion profile templates
• CLI etlantic report query and workspace file run history (.etlantic/history/)
• What's New / Migration / Exit Gate 0.34 documentation

Changed

• Official package versions align at 0.34.0; plugins require etlantic>=0.34.0,<0.35
• Terminal run reports may include metadata.observability event/delivery summary
• etlantic reliability quality-trends uses event consumer summary when --values omitted

Fixed

• Selected-engine PMPLUG402 now maps validate/plan CLI exit to TRUST_FAILURE (sibling discovery 402s remain non-blocking)
• Plugin registry descriptors retain package identity (distribution_name / package) after authorize-before-load
• Manual register_*_plugin refuses unauthorized overlays when a production profile is active
• Missing selected engines under production emit blocking PMPLUG404 trust diagnostics
• Plan snapshots with name=production require security_mode=production
• PySpark provider/plugin advertise Delta/merge capabilities only when delta-spark is importable; declared cache/checkpoint failures fail closed
• Medallantic missing run/step statuses fail closed; live IR scrubbing covers non-metadata secret keys; transform_ref imports support deny-by-default allowlists
• FileStateStore uses locked read-modify-write; schema/run history can fail closed on corrupt durable files
• Schema-drift / trust failures cannot be CONTINUE/SKIP'd; plan allow_trusted_sql is intersected with the live profile
• Polars/Pandas advertise only append/overwrite write extras; Airflow fails closed on non-cron/manual schedules; keyring capability ads match sync lookup; mounted-file secret errors omit absolute paths
• Structured run logs can fan out to observability providers via runtime bridge
• durable_audit fail-closed on run-history event append (not only terminal reports)
• File run-history JSONL append uses locked O_APPEND (cross-process safe; no full-file read)
• Run history query/list: orphan reports, idempotent create_run, UTC filter coercion, sorted report queries
• Production conformance requires run_history_provider when observability_delivery is durable_audit
• etlantic report query supports --since / --until; documentation drift for 0.34 M6 surfaces
• SQL temp-table materialization validates column identifiers (injection fail-closed)
• Asset URLs with userinfo credentials rejected from plan bindings (secret-free plans)
• Builtin allowlist exemption no longer trusts third-party engine/name spoofs
• Mounted-file secrets confined under mount root (path traversal fail-closed)
• Schema history scans field metadata, denies list-of-row bags, and uses collision-resistant subject paths
• Manual register_*_plugin overlays gated by production plugin_allowlist
• Non-blocking PMPLUG402 ignored consistently in trust exit, validation discovery, run/compile/doctor, and transform dispatch
• Quality membership fails closed on null; uniqueness skips null keys and rejected rows
• PipelinePlan.from_dict(verify=True) requires a non-empty fingerprint; production snapshot mode/name consistency checked
• Airflow artifact secret scan covers every strategy; plan snapshots keep only referenced secrets
• Airflow reference operator fails closed unless ETLANTIC_AIRFLOW_REFERENCE_STUB=1
• etlantic run maps validation/trust failures to documented exit codes; compile uses trust exit
• Nested pipeline definitions upgrade/validate to etlantic.pipeline/1
• Contract-diff toolkit exceptions fail closed as BREAKING drift
• --quiet still emits JSON/SARIF on trust gate failures
• Outbound emit payloads redacted in reports; authoring/redaction vocab includes pwd / AWS access keys
• Step TimeoutError is not retried under the default retry policy
• Spark cancel surfaces failures; PySpark advertises Delta only when delta is importable
• Strict production extension metadata rejects nested secret-like keys