Duckle v0.3.0

Duckle v0.3.0 - local-first, embedded ETL on DuckDB.

Patched since the initial v0.3.0 release

The binaries on this release have been refreshed. Re-download the binary for
your OS below to pick these up (the version number is unchanged).

2026-06-16 re-roll - workspace-relative paths, clearer auto-layout, materialization control, JSON zip

• Auto-layout now arranges the canvas by dependency depth (issue #36): nodes
  flow left to right by their position in the graph, siblings stacked and
  centered, with generous spacing so edges and connectors stay readable. The
  previous layout placed every node on a single row and ignored the wiring.
• Workspace-relative paths (issue #37): a built-in ${workspace} placeholder
  (alias ${projectroot}) resolves to the active workspace root, so source and
  sink paths can be written relative to it and a whole workspace folder stays
  portable when it is copied or moved. No context needs to be defined; it
  resolves on the canvas, in schema autodetect, and in headless / scheduled runs.
• Schema autodetect now resolves context variables before inspecting a source,
  so a path (or any field) bound to a context variable can be detected, not just
  a hand-typed literal. Previously autodetect sent the raw ${...} placeholder
  to the engine and could not find the file.
• Per-stage materialization control on every node's Basic tab. Choose how a step
  is stored: Auto (a view for a single consumer, a table when several steps read
  it), View (always lazy), Memory (read once, held as a table buffered in RAM),
  or Disk (read once, streamed through a temporary Parquet file to keep memory
  low for very large intermediates). In addition, a source that feeds a filter
  or quality validator which splits rows into pass and reject is now read once
  automatically instead of being scanned twice.
• New Zip Arrays to Table transform (Transform > Array). Turn a record that
  carries a list of column names and a list of row-arrays into a normal table
  with one column per name and one row per array - the common "headings + rows"
  JSON shape, with no hand-written SQL.
• The local webhook source no longer drops requests on macOS under load.

2026-06-15 re-roll - local accounts, faster context switching, corruption-safe workspaces

• Local multi-account profiles: create one or more named profiles (with an
  optional picture), shown in the top-right corner, and switch between them for
  quick context switching. Each profile remembers its own workspace folder, so
  switching profiles swaps the whole project context in one click. Profiles are
  stored only on this device, are never transmitted, and have no password -
  they separate working contexts, they are not a security boundary.
• A single corrupt workspace file no longer hides everything (issue #35):
  if a context, connection or pipeline JSON file is hand-edited into invalid
  JSON, Duckle now skips just that file and shows a banner naming it, instead of
  silently failing to load the entire workspace. If a structural file
  (duckle.json / repository.json) is invalid, the workspace stays
  un-editable and is never overwritten, so the good files on disk are protected
  until you fix or restore the broken one.
• Account-switching fixes (all part of the new profiles feature):
  • Switching to, or deleting back to, a profile that points at the workspace
    already open no longer blanks the canvas down to the default sample.
  • On a cold start Duckle opens the active profile's workspace, not the last
    globally used folder.
  • The account menu and editor are no longer clipped by the top bar, so the
    dropdown opens and you can switch, add, edit and remove profiles.
• Project: a documentation site is now live at
  https://duckle.org.

2026-06-14 re-roll (issue fixes)

• src.xml now captures text inside CDATA sections instead of skipping it, so a
  value written by snk.xml (which uses CDATA for complex cells) round-trips back
  correctly (issue #33).
• A pipeline run via a schedule now resolves workspace context the same way the
  canvas does, so a context-based value (for example an Oracle password stored
  as a context variable) is substituted before the run. Previously the raw
  placeholder reached the driver and a job that worked from the canvas failed
  under a schedule with errors like ORA-01017 (issue #32).

2026-06-13 re-roll (UI follow-up)

• The run-failure banner in the Output panel now has a dismiss (X) button on
  the right; it reappears on the next failed run.
• The component properties panel always opens on the Basic tab. Switching to
  Schema / Preview / Advanced / Validation is your choice and stays put while a
  component is selected; picking a different component resets back to Basic.

2026-06-13 re-roll - full-codebase correctness audit (67 fixes)

A multi-pass review of the whole codebase (engine, connectors, transforms,
desktop, frontend) with every finding independently verified before fixing.
Binaries refreshed; re-download below to pick these up.

• Reliability and data safety:
  • Cancelling a run now cancels only that run; a nested sub-pipeline (Iterate /
    ForEach / Run Job / Parallelize) no longer resets or steals another run's cancel.
  • The DuckDB engine and the local AI model now download atomically (written to
    a temp file and renamed into place), so an interrupted download can never
    leave a half-written file that looks installed.
  • A partial "Run from here" no longer advances incremental / change-feed
    watermarks, so a later full run cannot skip rows that a preview loaded but
    never wrote to a sink.
  • In fast batched runs, a failing transform is now blamed on the correct node
    instead of the downstream sink.
  • Autosave keeps a tab marked unsaved if the write actually failed, instead of
    silently losing edits.
• Formats and connectors:
  • Cloud (S3 / GCS / Azure) sources and sinks now reject Avro / ORC clearly
    instead of silently reading them as CSV or writing Parquet to a .avro/.orc path.
  • CSV "Windows-1252" encoding now works (it was previously rejected).
  • Kafka "Initial offset" (earliest / latest) is now honoured; the default
    reads the available backlog.
  • Snowflake and Databricks requests use the merged OS + bundled trust store
    again (fixes a corporate-proxy / Zscaler TLS regression). The Snowflake sink
    waits for an async statement to finish (no false success), REST page
    pagination starts from the right page, and the Snowflake source handles
    gzipped partitions and typed columns.
  • Identifier escaping hardened for ClickHouse, Cassandra and Oracle; the XML
    sink emits only valid element names; the Mongo source no longer drops a row
    when one value fails to convert.
• Transforms and SQL:
  • Window aggregate (aggwin) with an Order by keeps the per-partition total
    instead of silently becoming a running total.
  • INTERSECT / EXCEPT match by column name; NTILE uses the requested bucket
    count; rank direction is parsed correctly.
  • Denormalize, array-collect and JSON array-agg now produce a deterministic
    element order.
  • An aggregation on a named column with no function is rejected instead of
    silently becoming a row count.
• Write-mode safety:
  • A SQLite / DuckDB sink with an unrecognised write mode (a typo like "appnd")
    now errors instead of dropping and recreating the table.
  • Upsert with blank conflict columns is rejected; a database port above 65535
    is range-checked instead of wrapping to the wrong port.
• Hardening:
  • Closed several secret-leak paths in SQL export, the MCP server and git push
    output; the per-workspace secret key file is created owner-only.
  • UTF-8 panic guards in CSV type sniffing and the Map node, a size cap on the
    webhook body, temp-file cleanup, atomic config writes, and safer git
    clone/checkout argument handling.

2026-06-12 re-roll - secrets at rest, connector correctness, desktop reliability

• Secrets at rest: saved connection secrets (passwords, tokens, keys) and the
  cached git token are now encrypted with a per-workspace AES-256-GCM key under
  .duckle/keys/. Only secret fields are encrypted; host, database and user
  names stay readable. The key is gitignored, so a committed connections/
  folder can be shared without exposing credentials. Existing plaintext values
  are encrypted on the next save, and ${ENV:...} placeholders are never
  encrypted.
• Connector correctness:
  • Databricks sink no longer counts a still-running or failed write as
    success: it inspects the statement state, polls it to completion and fails
    loudly on error.
  • RabbitMQ source acknowledges messages only after the batch is durably
    written, so a write failure leaves them queued for redelivery instead of
    dropping them.
  • Webhook source answers 200 only after the batch is persisted (503 on
    failure), so a sender never treats a never-stored event as delivered.
  • The Avro sink infers a nullable schema, so a null in any column no longer
    aborts the whole file.
  • The XML sink splits a literal ]]> across two CDATA sections when writing
    nested values, keeping the output well-formed.
  • The MongoDB sink delete propagation matches boolean and numeric flag
    columns, not only strings.
  • Snowflake and Databricks delete propagation escapes backslashes in the
    delete value so it matches the source value.
  • The Snowflake source errors clearly on an unnamed result column instead of
    risking misaligned columns.
• Desktop reliability:
  • A cached git token that cannot be decrypted (missing workspace key) now
    reports a clear error instead of a confusing push-authentication failure.
  • The local model download integrity check no longer deletes a valid
    download on a transient read error.
  • Auto Layout now persists the new node positions.

2026-06-12 re-roll

• Engine correctness pass:
  • Cloud (S3 / GCS / Azure / HTTP) JSON sources now honour the Records path
    option and the 100 MB object-size cap, same as local JSON sources.
  • Inline dbt models with a non-identifier name (for example my-model) no
    longer fail with table-not-found on read-back; the written and read names
    now always agree.
  • A Distinct node with an Order by but no key columns now errors instead of
    silently dropping the ordering.
  • The GCP Pub/Sub source persists rows before acknowledging them, so a
    failure no longer loses the batch.
  • A REST response of {} now yields zero rows (like []), not one empty row.
  • BigQuery ATTACH escapes its project / dataset values.
• Multi-source guards and dbt:
  • Wiring a second input into a join / diff / SCD lookup port now errors
    (it was silently dropped). Map nodes still accept multiple lookups.
  • A dbt node exposes all its upstream tables to the project as the list
    var duckle_inputs (each is also a real table dbt reads via sources).
• App and community:
  • The native webview right-click menu (Back / Reload / Print) is suppressed
    on the app header, footer and chrome. The canvas and Projects tree keep
    their own menus, and text fields keep copy / paste.
  • A "Report a bug" button in the status bar and a Discord banner in the
    README link to the community server.

2026-06-11 re-roll

• Multiple DuckDB sources reading from the same database file in one pipeline
  no longer fail with database with name "duckle_src" already exists. Each
  attach-backed stage releases its alias so any number can target one file.
• History tab (#29): it rendered transparent so the canvas showed through;
  it now has an opaque background like the Run and Plan tabs.
• JSON source Records path (#27): the plain JSON source exposes the Records
  path option too, not just the JSONL source.
• Saved connections in node parameters (#30): a kind-filtered "Saved
  connection" picker at the top of each credential block auto-fills the
  fields below.
• Copy and paste components (#28): Ctrl/Cmd+C and Ctrl/Cmd+V copy and paste
  components within a pipeline or across pipelines.

Highlights

dbt, rebuilt around speed

• dbt Fusion is now the default engine: a Rust dbt runtime that parses and
  builds projects far faster than the Python toolchain. dbt Core (Apache,
  via dbt-duckdb) stays as an automatic fallback when Fusion is unavailable.
• Free first-launch provisioning: Duckle fetches and sets up its dbt engine
  for you on first use. No Python setup, no separate install, no cost.
• Multi-source dbt models: an xf.dbt node accepts several upstream inputs at
  once. Each wired source materializes a real table named by its node id, and
  your models read them through dbt sources - cross-system modeling inside
  one dbt build.
• A flagship Customer 360 example ships in the gallery: six sources across
  four system types feed a dbt build (staging, intermediate and mart models
  with tests), then enrich and fan out to four sinks.

JSON ingestion: nested record paths

• The JSON and JSONL sources have a Records path option. Point it at the key
  that holds the record array in a REST envelope (for example data, or a
  dotted path like response.records) and Duckle unnests it into proper
  columns, recursively flattening nested structs.

Native brand icons across the palette

• Sources, sinks and SaaS connectors show full-colour brand logos on the
  canvas, in the palette and in quick-add search. Transforms keep their
  themed glyphs.

Local multi-account profiles

• Create named profiles with an optional picture, shown top-right, each bound
  to its own workspace folder, and switch between them for quick context
  switching. Stored only on this device, never transmitted, no password.

Canvas and panels

• The right-hand properties panel is collapsible to the edge, with the state
  remembered between sessions.
• Each pipeline remembers its own canvas viewport (pan and zoom).

Operations (Phase 2)

• Structured error taxonomy in the engine for clearer, categorised failures.
• Prometheus metrics: a .prom textfile under the workspace logs/ dir for
  node_exporter's textfile collector or Grafana Alloy (no HTTP endpoint, no
  agent - works headless and air-gapped).
• Backfill controls: inspect and reset incremental watermarks from the UI and
  the runner.
• A Runs history tab to review past executions and trends.

Reliability

• A single corrupt or hand-edited workspace JSON file no longer blocks the
  whole workspace from loading; the bad file is skipped and named, and good
  files on disk are protected from being overwritten.
• All helper subprocesses (dbt, git, shell, the DuckDB CLI) run without
  flashing a console window on Windows.
• Engine staging is resilient when a binary is locked by a running instance:
  Duckle stages the update aside and swaps it in rather than failing with an
  access error.

Install

Download the raw executable for your OS below and run it - no installer.

Upgrading

This is a drop-in upgrade from v0.2.0. Existing pipelines and connections work
unchanged. The dbt engine provisions itself the first time you run a dbt node.