1.29.0

dlt 1.29.0 Release Notes

Highlights

• ClickHouse staging-optimized replace strategy (#3927 @filipesilva) — ClickHouse now supports the staging-optimized replace strategy, performing atomic table swaps via EXCHANGE TABLES so a full replace is instantaneous and never leaves the destination in a half-loaded state.
• AWS Secret Manager config provider (#4162 @rudolfix) — dlt can now resolve config and secrets directly from AWS Secret Manager, matching the existing Google Secret Manager provider. Also stops re-creating the Google secrets client on every lookup.
• Explicit joins in Relation.join() (#3868 @burnash) — The dataset relation API gains explicit join support, giving you control over join conditions when composing relations.
• Cross-destination join compatibility (#3905 @Travior) — Destination configs now expose a physical_location() accessor and can_join_with rules, so dlt knows when relations that live on different destinations (or datasets) can actually be joined together.

Core Library

Features

• BigQuery atomic replace (#4130 @oystersuki) — New opt-in enable_atomic_replace config flag makes the truncate-and-insert strategy do a single-job, metadata-preserving WRITE_TRUNCATE_DATA load from GCS staging, giving BigQuery an atomic, transactional full replace.
• DuckDB SQLAlchemy destination (#4161 @rudolfix) — Fully tests and fixes the SQLAlchemy destination running on DuckDB.
• Expose Parquet compression codec (#4183 @Travior) — Configure the Parquet compression codec via DATA_WRITER__COMPRESSION or ParquetFormatConfiguration(compression=...).
• Cross-batch schema evolution in ArrowToParquetWriter (#3896 @AyushPatel101) — Safe Arrow type promotions (e.g. float32 → float64) now work across flush batches instead of crashing once data spans multiple batches.
• Additional Redshift COPY options (#4088 @timH6502) — New Redshift config to append extra options (e.g. FILLRECORD) to the generated COPY command for staged loads.
• Cross-destination join compatibility (#3905 @Travior) — See Highlights.
• Disable dlt→dlthub command proxying in workspace context (#4037 @rudolfix) — Instead of silently running a dlthub command, dlt now shows a note and points to the correct dlthub counterpart, avoiding misleading behavior.

Fixes

• Fix: Normalize crash on empty Arrow tables with add_dlt_id (#4187 @nizar-zerrad) — Adding _dlt_id to an empty Arrow table no longer fails with ArrowInvalid.
• Fix: correct table/resource metric aggregation (#4155 @tetelio) — Writer metrics are now aggregated correctly when jobs for the same table interleave across normalize workers or file rotation (previously undercounted items_count).
• Fix: Airflow parallel staging truncation (#4079 @DRACULA1729) — Auto-disables staging truncation for decompose="parallel" / parallel-isolated runs so concurrent tasks no longer truncate each other's staging files.
• Fix: move misplaced merge subqueries to post-qualify (#4169 @rudolfix) — Corrects merge SQL generation where pre-qualify subquery collapse could not work correctly.
• Fix: switch replace strategy correctly (#4164 @rudolfix)
• Fix: use truncate mechanism for empty "replace" tables (#4153 @rudolfix) — Empty "replace" resources are truncated via load-package state rather than empty files, avoiding hint side-effects.
• Fix: use empty files to truncate top-level tables (#4168 @rudolfix) — Partially reverts #4153 for optimized replace strategies, where truncation happens upfront rather than inside the job transaction.
• Fix: reuse job client when truncating staging dataset (#4207 @rudolfix)
• Fix: keep the pipeline selector when no pipeline is selected (#4096 @lis365b) — Dashboard keeps the pipeline dropdown visible and shows a clearer empty-state message when the selection is cleared.
• Fix: logging more frequent than log_period (#3962 @cbility) — Stops resetting the log timer on every counter update, so logs respect the configured log_period.
• Bump jsonpath-ng and improve test coverage (#4105 @Travior) — Upgrades to jsonpath-ng 1.8.0, dropping the archived ply transitive dependency flagged by security scanners (CVE).

Docs

• Add Tigris to S3-compatible storage options for the filesystem destination (#4166 @davidmyriel)
• Add secrets management page to dltHub docs (#4124 @ShreyasGS)
• Add "Orchestrate with dltHub" cookbook entry (#4077 @ShreyasGS)
• Add Slack and email notifications guide (#4081 @ShreyasGS)
• Add playground workspace page (#4150 @AstrakhantsevaAA)
• Add playground destination page (#4148 @AstrakhantsevaAA)
• Clarify soft deletes description for MSSQL source (#4165 @AstrakhantsevaAA)

Chores

• Add git commit-message rule, hook, and /commit skill (#4098 @lis365b) — Claude Code tooling to keep commit messages Conventional-Commit clean (no footers/emojis).
• Refactor ibis handover — create ibis backend on the Destination factory (#4170 @rudolfix)
• Relax dlthub-client deps (#4202 @rudolfix)
• Fix Windows path for griffe tests (#4104 @rudolfix)

New Contributors

• @filipesilva made their first contribution in #3927
• @cbility made their first contribution in #3962
• @DRACULA1729 made their first contribution in #4079
• @oystersuki made their first contribution in #4130
• @davidmyriel made their first contribution in #4166
• @nizar-zerrad made their first contribution in #4187