Repository navigation
DuckDB 2.0 runs underneath now, and interval schedules run on the clock.
Read Before you upgrade first.
111 commits since v0.7.4.
Before you upgrade
-
Interval schedules run on the clock, in UTC. Every 15 minutes means :00,
:15, :30 and :45, and every 24 hours means midnight, so every interval
schedule fires at a new time, on one server as much as on several. -
DuckDB moves to 2.0.0. A
.duckdbfile Duckle creates still opens in
DuckDB 1.5. What you may notice, each measured against 1.5.5:- your own SQL:
x -> x + 1lambdas are refused (writelambda x: x + 1),
~matches anywhere in the string, and division by zero raises an error; - a merge sink fails on two source rows for one key instead of keeping the
first; - the first write to a DuckLake whose catalog is a DuckDB file upgrades the
catalog past what DuckDB 1.x can attach; - a Quack server still on DuckDB 1.5 cannot be reached.
The README's Moving to DuckDB 2.0 section lists every difference.
- your own SQL:
Several servers on one workspace
- Each schedule occurrence is claimed before it runs, by
serveand by the
desktop scheduler alike, so it runs once however many are looking. - A run accepted by one server can be followed and cancelled through another.
- What a dead replica was running is marked interrupted within about a minute,
instead of reading as running until a server restarts. - The servers have to share one host for now.
Test a pipeline where you build it
- Data Generator (
src.generate) makes believable fake rows: names,
emails, addresses, companies, phone numbers, network values, numbers and
dates. The same seed gives the same rows, so a pipeline can run without what
it connects to. - Tests tab, in the desktop app and the web editor: write test cases from
any node and run them through the same harness asduckle test. qa.assertstates a condition the rows must meet.
The canvas
- After a run, each node shows the rows it produced.
- Right-click the canvas for a note kept beside the nodes.
- Align selected nodes, and snap them to a grid.
- Double-click a node to open its editor, or the visual mapper for a Map.
- Generate documentation saves a pipeline as one self-contained HTML page,
with nothing secret in it. - Undo right after opening no longer writes the starter pipeline over your
file.
New nodes and options
- Kev on Classify. Kev, a Jev-like decision model you run on your own
machine, is a third engine on Classify: no API key for a local server, an
answer that is always one of your labels, and its probability beside it. - Nearest Join, DuckDB 2.0's
NEARESTas a node, and cast to VARIANT. - Delta: time travel on the source (
asOfVersion/asOfTimestamp) and
partitioned tables from the sink. - CSV, TSV and JSON Lines sinks append (#367).
- A REST source reads server-sent events (#365).
- A CSV source can detect whether its first row is a header.
src.changedchooses what a first run does with files already there,
filters by glob and modification time, takes a capped listing oldest first,
and no longer strands everything pastmaxEntrieson S3 (#324).- Run parameters bind to a schedule and to each pipeline in a plan step, and
editor runs check them like every other surface (#317). - A connection can be tested from its editor.
Green runs that were wrong
When two ran at once in one process:
- an FTP, FTPS or SFTP upload could send the other run's rows under a green
status; - two upserts with one upstream node id could corrupt each other's staging
table; - two runs of one Access read or dbt node shared temp files.
Each now has its own. And a .gz or .zst file sink published plain text
under the compressed name, a regression in 0.7.4.
Security
- Backfill and batch ids in the serve console are confined to their store
folders. - A run parameter an operator supplies can no longer carry SQL expression
syntax. connStringand a password written into a URL are treated as credentials
everywhere. MCP'sread_pipelineshows a typed secret as***and refuses
to write***back.- Dependency advisories fixed in russh, hickory, quinn-proto, rustls, zip,
pageant, zstd and others.
Also fixed
- Delete removes the item's file (#371).
- SQL Server autodetect works in the desktop app (#375).
- The first-run screen honours
DUCKLE_DUCKDB_BIN(#373). - ClickHouse strings through ADBC arrive as text (#364).
- A
ctl.diemessage names the offending row's columns (#366). - A zoned timestamp and a saved watermark no longer depend on the host's time
zone (#359). - A DuckDB found on
PATHstarts. duckle-runner --versionreports the release.