Duckle v0.7.1

Ten days and 220 commits since v0.7.0. The bulk of it is a legacy job importer that turns an existing Talend estate into Duckle pipelines, a REST fan-out that survives millions of parent rows and resumes where it died, and a set of boundaries that hold where the act happens rather than where it was planned.

Highlights

• A legacy Talend estate imports and runs, not just parses: 57 commits of expression translation, mapper semantics, loop bodies, sub-jobs and context resolution.
• A REST fan-out that survives two million parents, streams them instead of holding them, resumes from row 900,001 after a crash, and never buys the same row twice.
• A ceiling on the AI bill, plus a run that says out loud when it did not finish rather than reporting a partial dataset as complete.
• A reuse cache: a stage whose inputs did not change does not run again.
• DuckDB is now inside the network policy, and vault-held credentials resolve on every execution surface including MCP.

---

Legacy job import (Talend)

The importer went from "translates the shape of a job" to "translates the job". 57 commits, and most of them are a specific reading that turned a working step into a broken one.

Expression translation. Mapper expressions that read one way, the character helpers, numeric literals and sign-changing arithmetic, arithmetic written with signs, asking what a string contains, dates, counters, a value wrapped for its type, conditions and the shipped routines, a choice, a comparison. A string helper can take a count that is itself computed. A comma inside a literal is no longer treated as an argument separator.

Mapper semantics. A mapper's own named values and the ordinary string operations. Outputs are kept apart from each other. The condition deciding which rows reach a mapper output is preserved. A lookup is joined rather than dropped, travels with the loop it feeds, and stays qualified where the mapper looks something up. An output column with nothing to compute is still a column. A join key that is an expression counts as one key. A mapper written as name-to-expression is applied instead of passed through. Output types come from the declared schema, including exact decimals read by their type.

Job structure. A loop's body becomes a pipeline the loop can name. A reusable body is spliced into the job that calls it, keeping its boundary ports and the name of a named output port. Sub-flows are called, children and joblets resolve to the files they became, and work that waits for a subjob waits for all of it. Ordering links order the run without becoming data edges, never close a loop, and stay out of loop bodies. The order unlinked subjobs run in is kept.

Context and run variables. A context reference resolves from the job's own context and wherever else it appears. A Java body that only sets context values is carried over to nodes. A new component sets a run variable, and a run variable is carried into the jobs a pipeline runs. Context values are quoted where they stand for a value and kept out of the column namespace.

Refusing to guess. An untranslated Java body does not compile rather than silently doing nothing, and the report says which Java bodies carry no rules. A table written by more than one step is not mirrored. A query carrying a statement terminator is collapsed, and SQL that writes is reported.

Delimited files. Column names come from the declared schema, the quote setting is named the one the reader looks for, quoting off is stated rather than left to the reader's default, and whether a delimited writer emits column names is read from the job.

REST, at scale

• One request per row of an upstream table, with the response stamped onto the rows it produced.
• A fan-out that survives two million parents, and then streams the parent rows instead of holding them all in memory.
• Resume: a fan-out that died at row 900,001 continues from there. Each successful parent is recorded in the checkpoint store as it completes and replayed on rerun without reissuing the request. Never buy the same row twice.
• The incremental cursor reaches the request, so a fan-out asks for what changed. The cursor does not advance past a parent that failed under skip or reject, because advancing past rows that were never fetched loses them permanently.
• Keep the original response, named by its own content, and archive the raw response to object storage.
• Survives a rate limit, and sends requests in parallel.

AI nodes

• A ceiling on the bill. maxRequests, token and cost budgets, enforced with a compare-and-swap so the ceiling is exact under concurrency. A stop is not counted as a purchase.
• A run that says it did not finish. Hitting a budget marks the run incomplete rather than reporting a truncated dataset as a clean success, and that marker now reaches the run history, not just the CLI.
• Ask for a shape, and check the reply is that shape (structured output), with a pre-flight that refuses a schema field colliding with an upstream column, and expansion that refuses to overwrite the caller's own column.
• Checkpoint classify and embed, not only llm.

Reuse cache

• A stage whose inputs did not change does not run again. Opt-in, keyed on the stage config, its input fingerprint and the engine version.
• An engine upgrade invalidates the cache, because the same inputs can produce different output from a different engine.
• See, drop and distrust the cache from the CLI.

Python

• Use the workspace's own virtual environment rather than whatever interpreter was on PATH.
• Refuse a run whose .venv is not the environment uv.lock describes. A target that declares a locked environment and has nothing installed is refused rather than run against the wrong packages.
• Hand a script the whole table, through Parquet, and stream a table through code.python instead of materializing it.

Documents and artifacts

• Read the schema the feed already published: point src.xml at an XSD and the columns are derived from it, with the exact schema bytes recorded in the signed run manifest.
• A corpus from an upstream relation, not one configured path, for XML, HTML and PDF.
• HTML: read rows out of a page by CSS selector, and follow the pagination a server rendered. A walk cut short by a failed page now reports as incomplete rather than as a clean run.
• PDF: one row per page.
• Archives: unpack an archive into artifacts; a skipped member keeps the identity it was written with.
• Artifacts describe files as rows a pipeline can reason about, carry business keys through a parser, and copy the bytes while recording where they came from.
• Bounded intermediates for XML, and a bounded corpus list rather than only bounded documents.

Streaming and continuous runs

• follow, a continuous mode for pipelines that track their position.
• listen + src.spool, so a push source stops losing what arrives.
• src.changed: poll a remote source, including S3 and S3-compatible stores, without downloading it. A source that found nothing reports as unchanged rather than a plain ok, and unchanged is persisted and exposed as its own metric.
• xf.tumble, event-time tumbling windows that survive between runs.
• Kafka: resume where the last successful run stopped, decode Confluent-framed Avro, and apply the security settings the form has always offered.

Quality and testing

• duckle test: assert what a pipeline produces from a fixed input. Row count, uniqueness, not-null, a SQL predicate, numeric tolerance, column types rather than only rendered values, DECIMAL precision when the expectation spells it out, and deterministic ordering for a case that never promised one. A fixture lives beside the test file.
• A baseline: let an operator say this is the new normal.
• Propose only the record pairs worth comparing.
• A quality gate told to fail the run now actually fails it.
• run_tests over MCP, so an agent can run the suite without a shell.

Security, policy and secrets

• The policy boundary is enforced where the act happens, not only where it is planned, so a URL built at run time or a redirect hop meets the same refusal.
• DuckDB is inside the network policy. Under mode: enforce with a network.allowedDomains list, a run starts with DuckDB's remote filesystems disabled, community extensions refused and extension autoinstall off. Two of those settings are one-way in DuckDB, so generated SQL cannot lift them.
• Fetch a credential from an external vault at run time with ${VAULT:NAME}, and, as of this release, on every execution surface: the MCP server previously ran pipelines without resolving it, so a vaulted credential arrived at the connector as literal text.
• SFTP remembers host keys, so an unpinned connection notices a change.
• A run that cannot record its state no longer reports ok.

Connectors and sinks

• Neo4j, Turso/libSQL and IBM DB2.
• GeoParquet from the Geospatial sink.
• Snowflake: a writeMode that replaces the table instead of adding to it.
• DuckLake: maintain a lake through the same pipelines that fill it, and publish several tables as one snapshot or refuse.
• A typed file operation, a listing that answers "is it there", rows written in place, and a folder listing.
• Set a proxy, timeouts and a User-Agent once, in one place.

Server, backfill and the console

• Accept a run, answer for it later, and cancel it.
• Headless backfill across the console API, MCP and the web editor, and a backfill set can no longer destroy other state. A run's flush no longer overwrites an edit made during it.
• A resource budget, so one job cannot take the machine down.
• Bound how long a queued item is retried.

Correctness fixes worth calling out

Several of these are the silent-data class, where the run reports success and the data is wrong:

• Describing a node in the editor ran the node's SQL, which for a sink meant executing its COPY ... TO against an empty stub and truncating the user's real output file on a click.
• A JSON column that appears late is no longer lost.
• Flatten now actually flattens, and repeated keys keep their parent.
• GCS region and the rest of a bucket's settings are carried into the secret.
• A headerless CSV takes its column names from its declared schema.
• A model is registered only when the run succeeded.
• XML entities are no longer dropped (quick-xml 0.42 splits them into their own events).

Dependencies

21 commits of dependency work: axum 0.8, rusqlite 0.40, apache-avro 0.22, lapin 4, aes-gcm 0.11, ed25519-dalek 3, zip 8, redis 1.6, jsonwebtoken 11, suppaftp 10, getrandom 0.4, prost 0.14, odbc-api 29, quick-xml 0.42, toml 1.1, scylla 1, jaq 3, the arrow family onto one major, russh 0.63, TypeScript 7, Vite 8. Security advisories reaching the shipped binary and the frontend build tooling are patched. The encrypted-secret format was proved to survive the aes-gcm and ed25519 majors rather than assumed to.

---

Upgrade notes

One behaviour change can break a working pipeline, deliberately. If your policy sets network.allowedDomains and mode: enforce, DuckDB itself is now taken off the network for the run. A pipeline that relied on DuckDB reading https:// or s3:// directly under such a policy will now fail closed rather than quietly bypassing the allowlist. Two ways forward:

• route the read through a Duckle connector, which enforces the allowlist per host; or
• set network.allowDuckdbExternalIo: true in the server policy, which returns DuckDB-native remote reads along with the boundary they cost. It widens the boundary, so a workspace policy can never set it.

Local file access is untouched.

The MCP server now resolves ${VAULT:...} and ${ENV:...}. A pipeline that had been silently receiving the literal placeholder will now receive the real value. If anything downstream depended on the literal text, it will change.

Verification

Full suite green before the tag: 707 engine unit tests, 417 execution tests, 162 runner, 19 MCP, 21 secrets, 11 scheduler. Engine tests run against the pinned DuckDB 1.5.4.