Skip to content

v1.11.13

Choose a tag to compare

@github-actions github-actions released this 11 Apr 14:39
· 11 commits to master since this release

Features

  • External materialization — write dbt models directly to Parquet,
    CSV, or JSON files via server-side COPY statements. Because GizmoSQL
    is remote DuckDB, the COPY runs on the GizmoSQL server (typically a
    cloud VM with more CPU, memory, disk throughput, NIC bandwidth, and
    cloud-IAM reach than the dbt client) rather than streaming result sets
    back to the client just to write them out. Supports:
    • Local filesystem paths and any URI the server's DuckDB backend can
      reach: s3://, gs://, azure://, MinIO and other S3-compatible
      stores, etc.
    • Format inference from file extension, explicit format config, custom
      delimiter, and arbitrary DuckDB COPY options (compression
      codecs, partition_by, per_thread_output, ...).
    • Default {external_root}/{model_name}.{format} locations via a new
      external_root profile setting (resolved on the server).
    • ref()-able — a view is created over the written file so downstream
      models can use it like any other relation.
  • New adapter-level @available helpers: external_root,
    external_write_options, external_read_location, location_exists,
    store_relation, warn_once.
  • New Jinja macros: materializations/external.sql and
    utils/external_location.sql.
  • plugin / glue_register options (client-side features in dbt-duckdb)
    produce a clear compile-time error — they have no analogue in a
    server-side Flight SQL adapter.
  • README: new "Writing to External Files (server-side)" section with the
    config table, profile example, partitioning example, and notes on the
    parent-directory requirement for local file writes.

Test suite

  • New tests/functional/adapter/test_external.py (7 test classes):
    default parquet, CSV, JSON, explicit .parquet location, explicit
    .csv location + custom delimiter, downstream ref(), empty-result
    handling, plugin/glue rejection, parquet compression codec, hive-
    partitioned writes, and an end-to-end S3 test using a MinIO sidecar on
    a user-defined docker bridge network.

Full Changelog: v1.11.12...v1.11.13