Skip to content

v0.6.0

Latest

Choose a tag to compare

@dannote dannote released this 26 Oct 17:31
· 7 commits to master since this release

QuackDB now targets DuckDB 2.0 and its stable Quack extension, which speaks Quack protocol version 3. DuckDB 1.5 servers are no longer supported: the two protocol versions are not wire-compatible, and connecting to an older server fails with :unsupported_quack_version.

Breaking changes

  • Requires DuckDB 2.0. The managed binary is DuckDB 2.0.0, downloaded as the new duckdb-cli-<target>.tar.gz archives with checksums for all Linux and macOS targets, including osx-universal. The DuckDB 1.5 checksums are gone, and QuackDB.Binary rejects CLIs older than 2.0.
  • QuackDB.Server no longer sets the removed quack_fetch_batch_chunks setting, which made DuckDB 2.0 exit during boot (#8). DuckDB 2.0 sizes result batches by bytes with quack_target_batch_bytes.
  • Native appends no longer use the removed APPEND_REQUEST message. Each append is one INSERT INTO table (columns) SELECT * FROM scan_data_from_quack_client(...) statement fed by numbered send-data batches. Omitted columns now get their defaults and sequence values, and append errors no longer carry the Failed to append: prefix.
  • QuackDB.Profile follows DuckDB 2.0's grouped profile JSON: query, parser, planner, physical_planner, optimizer, io, system, and operators. Operators carry type, timing, intermediate_rows, and rows_scanned; the flat 1.5 fields such as latency and operator_timing are gone.
  • Append type inference considers every row and value instead of the first. Rows with keys the first row lacks now append those columns too, and columns that mix shapes, such as maps and integers, fail with :mixed_append_types instead of dropping or NULLing values. Declare :columns (or :variant) for such data.
  • result_uuid is renamed query_uuid in QuackDB.Query, QuackDB.Cursor, result metadata, and fetch telemetry, matching the protocol.
  • DuckDB 2.0 itself changes some output: function-call column names follow the new parser ("version"()), integer division by zero raises, EXPLAIN renders boxes with readable operator names, and database files written by 2.0 cannot be opened by DuckDB 1.5.

Added

  • Query cancellation. QuackDB.cancel/3 interrupts the query on a remote connection from another process, and the query fails with :query_cancelled. A transport timeout cancels its query on the server, and DBConnection disconnects cancel over a fresh HTTP connection.
  • Typed server errors. QuackDB.Error metadata carries DuckDB's :exception_type (also via QuackDB.Error.exception_type/1) and transferable :extra_info. A server whose database died returns :connection_invalidated and an expired heartbeat lease returns :connection_expired; both disconnect so the pool reconnects.
  • Heartbeat leases. Connections request a lease with :heartbeat_timeout (60 seconds by default) and renew it every third of the lease over a separate HTTP connection, as DuckDB's own client does, so queries, streams, and transactions that outlast the lease keep their connection. DBConnection idle pings send heartbeat messages instead of SELECT 1. :client_id names the client in server logs.
  • ON CONFLICT and RETURNING for native appends: insert_rows/4 and insert_columns/4 accept :on_conflict and :returning, and Ecto insert_all(..., insert_method: :append) runs on_conflict: :nothing and returning as one statement instead of staging rows in a temporary table. Schemaless appends can use returning.
  • VARIANT decoding, unshredded and shredded, into Elixir terms. :variant and :json append column types send JSON text that DuckDB casts, so mixed objects, arrays, and scalars keep their shapes.
  • QuackDB.Ecto.Variant with variant_extract/2 (keys, 1-based indexes, and paths), variant_extract_string/2, variant_type/1,2, variant_typeof/1, variant_keys/1,2, variant_array_length/1,2, and variant_contains/2, imported by use QuackDB.Ecto.
  • TUPLE values decode to Elixir tuples, and TIMESTAMPTZ_NS values to QuackDB.NanosecondTimestamp.
  • Ecto json_set/3, json_insert/3, json_replace/3, and json_remove/2, and variable/1 for DuckDB 2.0's $name variables, with QuackDB.SQL.set_variable/2 and reset_variable/1.
  • QuackDB.DDL.create_trigger/4 and drop_trigger/3 for DuckDB 2.0 triggers, and create_schema/2 and drop_schema/2. Table names accept nested schema paths such as {[:finance, :reports], :ledger}, including the :schema append option.
  • Ecto migrations can execute QuackDB.DDL builders directly (execute DDL.create_trigger(...)); the adapter accepts iodata DDL.
  • Reconnect support: acknowledge_results: true confirms each full result to servers running with quack_enable_reconnects, and a fetch that lost its HTTP connection requests the same numbered batch again (:fetch_retries, default 1).
  • An :inline_rows query option caps the rows of the first response; cursors default it to :max_rows.
  • A :copy_binaries option, per connection or per query, copies decoded strings and blobs off the HTTP response, so keeping a few values from a large result no longer keeps the whole response in memory.
  • QuackDB.Server accepts :serve_options (allow_other_hostname, disable_ssl, ssl_cert_file, ssl_key_file) and derives an https URI for non-local hosts as quack_serve does. The CLI runs with -no-agent so DuckDB 2.0's agent output mode stays off under coding agents.

Changed

  • {:json, term} parameters encode with Elixir's built-in JSON module, so they no longer need Jason.
  • Faster result decoding. Fixed-width vectors decode with one binary comprehension per column instead of one call per value, temporal values are built from calendar fields, and LIST, ARRAY, STRUCT, and VARCHAR vectors decode in one pass. Per 2048-row vector this takes 2x fewer reductions for integers, lists, and arrays and 4x fewer for timestamps.

Fixed

  • Connecting to DuckDB 2.0 failed with unknown error response field 2 (#8).
  • Dictionary, ARRAY, and VARIANT vectors decoded in quadratic time: a 2048-row dictionary vector took 1.2 ms instead of about 60 µs, and a 4000-element VARIANT array 29 ms instead of about 0.5 ms.
  • Empty STRUCT values, which DuckDB 2.0 allows (SELECT {}), failed to decode.
  • GEOMETRY columns with a coordinate reference system, such as GEOMETRY('OGC:CRS84'), failed to decode with :invalid_string.
  • Append type inference looked only at the first non-nil value of each column. A later larger integer was silently truncated to 32 bits, struct keys and row keys missing from the first value were dropped, a non-map value in a struct column became NULL, and integers mixed with floats or strings crashed while encoding. Inference now considers every value: integers widen to BIGINT or HUGEINT, integers join floats as DOUBLE while that stays exact, decimals widen to one DECIMAL, and struct fields and row columns are the union of all keys. Any other mix fails with :mixed_append_types, naming the column and suggesting :columns or :variant.
  • DECIMAL(p, 0) values decoded as integers instead of Decimal, because DuckDB omits a zero scale.
  • Result types QuackDB cannot decode, such as UNION, and malformed UTF-8, BIGNUM, and BIT values raised and killed the connection process. The query now fails with :unsupported_type or the matching decode error.
  • A failed handshake (a wrong token, an older server) left its HTTP connection process running, one more on every reconnect attempt.
  • Ecto reads, update_all, and delete_all ignored schema prefixes and queried the unqualified table: from(... prefix: ...), @schema_prefix, and Repo.all(query, prefix: ...) now qualify the source, as inserts and migrations already did. A dotted prefix such as "finance.reports" names a nested schema or catalog.schema, in reads and in insert_method: :append. As in Ecto's planner, the query prefix does not apply to CTE names.