v1.11.13
·
11 commits
to master
since this release
Features
- External materialization — write dbt models directly to Parquet,
CSV, or JSON files via server-sideCOPYstatements. Because GizmoSQL
is remote DuckDB, theCOPYruns on the GizmoSQL server (typically a
cloud VM with more CPU, memory, disk throughput, NIC bandwidth, and
cloud-IAM reach than the dbt client) rather than streaming result sets
back to the client just to write them out. Supports:- Local filesystem paths and any URI the server's DuckDB backend can
reach:s3://,gs://,azure://, MinIO and other S3-compatible
stores, etc. - Format inference from file extension, explicit
formatconfig, custom
delimiter, and arbitrary DuckDBCOPYoptions(compression
codecs,partition_by,per_thread_output, ...). - Default
{external_root}/{model_name}.{format}locations via a new
external_rootprofile setting (resolved on the server). ref()-able — a view is created over the written file so downstream
models can use it like any other relation.
- Local filesystem paths and any URI the server's DuckDB backend can
- New adapter-level
@availablehelpers:external_root,
external_write_options,external_read_location,location_exists,
store_relation,warn_once. - New Jinja macros:
materializations/external.sqland
utils/external_location.sql. plugin/glue_registeroptions (client-side features in dbt-duckdb)
produce a clear compile-time error — they have no analogue in a
server-side Flight SQL adapter.- README: new "Writing to External Files (server-side)" section with the
config table, profile example, partitioning example, and notes on the
parent-directory requirement for local file writes.
Test suite
- New
tests/functional/adapter/test_external.py(7 test classes):
default parquet, CSV, JSON, explicit.parquetlocation, explicit
.csvlocation + custom delimiter, downstreamref(), empty-result
handling, plugin/glue rejection, parquet compression codec, hive-
partitioned writes, and an end-to-end S3 test using a MinIO sidecar on
a user-defined docker bridge network.
Full Changelog: v1.11.12...v1.11.13