Skip to content

v0.20.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Oct 17:45
· 203 commits to main since this release
v0.20.0
c60003b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v0.20.0 is the first stable release since v0.19.0 (2026-07-24). It adds conduit generate, which
writes a pipeline config from a plain-English description; a Postgres-to-pgvector RAG pipeline
template with new chunking and embedding processors; outbound HTTP for WASM processors, closed by
default; and signed release artifacts. It fixes a SEV-0 bug, present in v0.19.0, where a Postgres
snapshot-to-CDC handoff could stall silently, and a gRPC server panic. It also changes how stopped
pipelines are reported and recovered; read Behavior changes before you
upgrade. The default pipeline engine is unchanged. The RAG pipeline needs the preview engine
(arch-v2).

Upgrading from v0.19.0

Upgrade if you run v0.19.0:

  • SEV-0 snapshot-to-CDC stall. In v0.19.0 the Postgres source could stop after its snapshot
    without starting CDC. Inserts, updates and deletes after the snapshot were not delivered, there
    was no error, and the pipeline still showed running. Fixed in this release.
  • gRPC advisory GO-2026-6443. v0.19.0 ships grpc v1.82.1, which can panic on a request with no
    :authority or Host header. v0.20.0 ships v1.83.2.

Check these before you upgrade:

  • Stop and shutdown status. A force stop is now UserStopped, not Degraded, and no longer
    triggers pipelines.exit-on-degraded. Stopped pipelines can carry an error message. Pipelines
    stopped by a shutdown are SystemStopped and start again on the next boot. If you alert on
    Degraded or on status alone, see Behavior changes.
  • Error text. Fan-out on the default engine now fails with pipeline.fanout_requires_arch_v2,
    and Start during shutdown fails with pipeline.shutting_down. Update anything that matches on
    error text.
  • Community links. The Discord invite is gone. Use
    GitHub Discussions. Docs links now point to
    conduitdata.io; conduit.io no longer resolves.
  • Mirrors. If you mirror releases, also copy checksums.txt.sigstore.json and the
    .sbom.json files.

Highlights

conduit generate (preview)

conduit generate "<description>" writes a pipeline config file using Anthropic, OpenAI or a local
Ollama server. Set ANTHROPIC_API_KEY or OPENAI_API_KEY, or run Ollama, and pick one with
--provider if more than one is available. Every candidate goes through the same offline check as
conduit pipelines validate, including that each connector it names exists, and the command never
writes an invalid config. It does not deploy anything: there is no --apply flag, and deploying is
the usual conduit pipelines deploy step.

The command is a preview. Review what it writes before you deploy it. On our 28-request test
corpus with Claude Sonnet 5, every generated config validated, and about two in three (64–68%
over three runs) matched what the request asked for. It is reliable at picking connectors
(45/45 on requests that need no processor) and often leaves out a requested processor step (8/39
on requests that need one). These are measurements from one model on a small corpus. A weekly
eval tracks them from v0.20.1, and processor-aware generation is planned for v0.21.

Correction (2026-10-12): the semantic-match and processor figures above understate
conduit generate. Our scorer did not accept processor names written with the builtin: prefix,
which is the form the prompt tells the model to use, so some correct pipelines were scored as
missing a processor. Rescoring the same recorded outputs with the fixed scorer moves the corpus
from 18 to 22 of 28 matched requests. The validation result is unaffected. Corrected numbers,
measured on fresh runs, will be in the v0.20.1 release notes.

conduit generate "read from postgres and write new rows to s3 as json" --out pg-to-s3.yaml
conduit pipelines validate pg-to-s3.yaml

--json is supported.

RAG pipeline template (requires the preview engine and egress)

The postgres-pgvector-rag template reads Postgres (snapshot, then CDC), splits text with
ai.chunk, embeds it with ai.embed, and upserts into pgvector. Writes are keyed so redelivery
and updates converge to one row per chunk, and deleting a source row deletes every chunk made from
it.

It needs two things the defaults don't give you:

  1. The preview engine, --preview.pipeline-arch-v2. ai.chunk turns one record into many,
    and the default engine passes one record in and one out. On the default engine the pipeline
    stops at the chunk step with pipeline.fanout_requires_arch_v2.
  2. The egress ceiling. WASM processors have no network access unless the operator allows it.
    The template already sets ollama.baseURL: http://127.0.0.1:11434 and
    sdk.egress.allow: http://127.0.0.1:11434 on the embed processor. You open the ceiling with
    --processors.egress.enabled --processors.egress.allow http://127.0.0.1:11434, or
    processors.egress.enabled: true and processors.egress.allow: [http://127.0.0.1:11434] in
    conduit.yaml. Without it every embed call fails with
    http egress is not enabled for this processor and the record is nacked.

Use the IP literal. The egress gate rejects http://localhost:11434 as an allowlist entry and
only admits loopback through an exact IP:port entry. If you scaffolded this template from a
nightly up to and including v0.20.0-nightly.20261007, change ollama.baseURL to http://127.0.0.1:11434 and add
sdk.egress.allow: http://127.0.0.1:11434 to the embed processor, or scaffold it again.

conduit pipelines init --template postgres-pgvector-rag   # prints the prerequisites
conduit connectors install pgvector
conduit processor-plugins install ai.chunk
conduit processor-plugins install ai.embed
conduit run --preview.pipeline-arch-v2 \
  --processors.egress.enabled --processors.egress.allow http://127.0.0.1:11434

Create the pgvector table first; the template README has the SQL and the settings for a hosted
provider such as OpenAI. ai.chunk and ai.embed (0.1.0) require Conduit 0.20.0 or later.
conduit-connector-pgvector v0.1.0 is a standalone connector, not built in.

Registry installs depend on a fresh signature on the hosted index. The client refuses an index
signed more than 7 days ago with registry.index_stale. For offline or air-gapped hosts, use
--index-file (a local signed index) or --bundle (a signed bundle).

The flag is temporary. Conduit is moving to a single pipeline engine that supports one-to-many
processors, fan-out and multiple sources and destinations, so you won't have to choose. Until that
engine is the default, the RAG template needs --preview.pipeline-arch-v2. Once it is, the flag
will be accepted and ignored with a deprecation warning before it is removed. We make no
performance claims for arch-v2 and don't recommend it for pipelines that don't need fan-out.

Host egress for WASM processors (processor SDK v0.6.0)

Standalone (WASM) processors can make outbound HTTP calls through the host. ai.embed uses this
to reach its embedding provider. Egress is closed by default. A processor gets it only when the
operator opens the instance ceiling (processors.egress.*) and the pipeline opts that processor in
with sdk.egress.allow; the effective allowlist is where the two overlap. Addresses are checked
again when the connection is made, so a hostname that resolves to a private, link-local or
metadata address is refused. Credentials stay on the host: grant a name in
processors.egress.secret-refs, export CONDUIT_SECRET_<NAME>, and the host injects the
Authorization header. The secret never enters the guest. Build processors against
github.com/conduitio/conduit-processor-sdk v0.6.0. Older processors work unchanged.
Configuration reference: README, "Processor host egress".

conduit processor-plugins install / uninstall

Installs WASM processors from the signed registry, with the same signature and SLSA provenance
checks as conduit connectors install. Offline installs work with --index-file or --bundle.
uninstall refuses to remove a processor that a pipeline still uses.

conduit processor-plugins install ai.chunk --dry-run --json

The registry index now lists processors next to connectors. The change is additive, so older
clients ignore processor entries and keep installing connectors.

Signed releases

From this release on, every release and nightly is signed with Sigstore keyless signing through
GitHub Actions OIDC. There is no long-lived key. checksums.txt is signed
(checksums.txt.sigstore.json), and since it lists every archive, package and SBOM, one signature
covers all of them. Each archive comes with an SPDX SBOM (<archive>.sbom.json). The Docker image
on GHCR is signed by digest. See the snippet at the end of these notes or
"Verifying a release".
Releases up to and including v0.19.0 are not signed. macOS binaries are not notarized yet.

CI gates

The chaos suite (tests/chaos) is a required check on main and runs on every PR and nightly. It
SIGKILLs pipelines mid-snapshot, mid-batch and mid-checkpoint, then checks for loss and position
correctness on recovery. govulncheck runs on every PR, push to main and nightly, and fails a PR
that makes a new vulnerability reachable. bundle-e2e (WASM egress) and rag-e2e (chunk, embed,
pgvector) are also required.

Behavior changes

Stopped pipelines are never recovered (#2901)

Both engines. Details:
ADR 20261007.

  • Once a stop is requested, by you or by shutdown, the pipeline is not sent into recovery and not
    restarted.
  • A user Stop, graceful or force, ends UserStopped. If the drain hit an error, the error is
    kept.
  • A force stop is now UserStopped with force stop recorded. It was Degraded. It no longer
    runs OnFailure handlers, so it no longer shuts Conduit down under pipelines.exit-on-degraded.
  • A shutdown, graceful or not, ends SystemStopped with any error kept. Those pipelines start
    again on the next boot and resume from their last persisted position.
  • Under exit-on-degraded, only the pipeline whose own fatal error triggered the exit is
    Degraded. The other pipelines are SystemStopped with the reason recorded and auto-start next
    boot. Before, they were also Degraded and needed a manual start.
  • Start and recovery restarts during shutdown are refused with pipeline.shutting_down (gRPC
    Unavailable).
  • A shutdown during a recovery backoff completes at once instead of waiting out the backoff or the
    30s exit timeout.
  • Drain errors are kept: shutdown returns them as its error, and arch-v2 now records them on the
    pipeline.
  • A stopped pipeline can carry a non-empty error. It means the drain did not end cleanly; the
    affected records were nacked and replay on the next start. Dashboards that alert only on
    Degraded no longer see drain failures. Alert on the error field of stopped pipelines too.
  • pipeline_recovering_count no longer counts a run whose stop was requested before it failed.
Case v0.19.0 v0.20.0 Error kept OnFailure
User Stop, clean drain UserStopped UserStopped — no
User Stop, error during drain default engine: recovered and restarted; arch-v2: UserStopped, error dropped UserStopped yes no
User force stop Degraded UserStopped force stop no (was yes)
User Stop during recovery backoff restarted after the backoff UserStopped original error no
Shutdown, clean drain SystemStopped SystemStopped — no
Shutdown with an error (drain error, runtime error, exit-on-degraded) default engine: restarted during shutdown, or Degraded if the cause was fatal; arch-v2: error dropped SystemStopped, auto-starts next boot yes no (was yes when Degraded)
Fatal error before any stop Degraded Degraded yes yes
Transient error, no stop Recovering, then restarted unchanged — —
Start during shutdown could start refused, pipeline.shutting_down — —

Other changes

  • Built-in connectors: postgres v0.14.0 → v0.14.2, kafka v0.12.3 → v0.12.5, file v0.10.3 →
    v0.10.8, log v0.7.3 → v0.7.5. Dependency updates and re-releases with signed artifacts, no
    feature changes.
  • New error code pipeline.fanout_requires_arch_v2 replaces unknown record type: sdk.MultiRecord when a fan-out processor runs on the default engine.
  • --preview.pipeline-arch-v2 still defaults to off. Its help text now says what it is for
    and names one limit: a split record whose pieces are cut across a multi-destination fan-out fails
    with pipeline.split_run_straddles_fanout.
  • New processors.egress.* settings (enabled, allow, secret-refs, timeout,
    max-response-bytes). With the default enabled: false nothing changes. Processor settings
    under sdk.egress.* are reserved for the host and are no longer passed to the processor.
  • Settings are redacted in API responses. Connector, processor and DLQ setting values come
    back as "***". Requests must send real values (see Known issues, #2913).
  • Embedded Conduit with the API enabled (#2747): /metrics serves from the registry you pass
    in. A MetricsRegisterer that is not also a prometheus.Gatherer now makes the first Run or
    Import return CodeInvalidArgument; before, it started and served unrelated default metrics.
  • Prerelease builds and minConduitVersion (#2822): a v0.20.0 nightly now satisfies a
    minConduitVersion of 0.20.0.

Fixes

  • SEV-0: Postgres snapshot-to-CDC handoff could stall silently (#2707). Present in v0.19.0. An
    ack the source waits for at the end of a snapshot could be dropped under load, so CDC never
    started and later changes were never delivered, with no error and no DLQ entry. Found before
    release by the RAG e2e; no production occurrence is known. Postmortem:
    docs/postmortems/20260729-snapshot-handoff-deferred-ack-deadlock.md.
  • Connector state write errors were ignored, so a source could ack upstream without a saved
    position (#2925).     Present since 2023, including v0.19.0. If saving one connector's state failed
    inside an otherwise healthy write, the error was only logged, the write was committed without it,
    and the source was told the position was saved. It then released its upstream ack (for example a
    Postgres slot advance or a Kafka offset commit), so after a restart those records could be lost.
    The failure now aborts the write and fails the pipeline instead.
  • gRPC server panic, GO-2026-6443 / CVE-2026-84445 (#2881). A request with no :authority or
    Host header could panic the gRPC API server. v0.19.0 ships the affected grpc v1.82.1; v0.20.0
    pins the fixed v1.83.2.
  • Go standard library advisories (#2914). Moving to Go 1.25.14 clears the 20 standard-library
    advisories govulncheck found reachable in the Conduit binary.
  • HTTP/2 server advisories GO-2026-6603, -6611, -6612, -6617 (#2970). A malicious HTTP/2 client
    could crash the API server, exhaust its memory or CPU, or corrupt flow control. v0.20.0 builds
    with Go 1.26.9 and golang.org/x/net v0.60.0, which fix all four. Go 1.25 gets no fix upstream, so
    building Conduit from source or embedding it now requires Go 1.26 or newer.
  • Shutdown could restart pipelines and write positions after the database closed (#2901). On a
    non-graceful shutdown, or a transient error during a user Stop, the default engine sent
    pipelines into recovery and restarted them. During shutdown, a restarted run could write
    positions after the database was closed. Fixed by the stop semantics above.
  • Failed pipeline reported as UserStopped (#2896). Default engine. A node error could leave a
    pipeline marked user-stopped with no error, skipping OnFailure and recovery, so a transient error
    stopped the pipeline for good. No data loss.
  • arch-v2: run cleanup could erase a live run (#2811). If a status write failed during a
    recovery restart, the superseded run's cleanup could remove the live run, which then could not be
    stopped or waited on.
  • Process panic on shutdown of a multi-source pipeline (#2749). Shutting down a pipeline with
    two or more sources could crash Conduit with sync: WaitGroup is reused. Reproduced on v0.19.0.
  • Recovered pipeline reported with its old error; shutdown could close the DB under a live run
    (#2807, #2812).     Default engine, on recovery restarts.
  • conduit init then conduit run failed out of the box (#2855). The generated conduit.yaml
    wrote empty lists as the string '[]', which conduit run rejected.
  • conduit pipelines init with no arguments failed outside a workspace (#2857). It now creates
    pipelines/, and remaining failures return a validation code with the OS error and a fix.
  • processor-plugins install refused every processor (#2818). Fixed before any stable release
    shipped the command.
  • Other arch-v2 fixes: destination-scoped processors run before the write (#2805); split records
    hold back their source position until every piece is done (#2731); nack and retry bounds (#2732,
    #2733); empty batch and empty position guards (#2738, #2744).

Known issues

  • arch-v2: destination batching can stall (#2979). On --preview.pipeline-arch-v2, a destination
    with sdk.batch.size above 1 and the default sdk.batch.delay of 0 can stop delivering, and with a
    delay set it slows to roughly one batch delay per record. The defaults and the RAG template are not
    affected. On arch-v2, don't set sdk.batch.size on destinations. v0.20.1 refuses that
    configuration at start with an actionable error.
  • Shutdown can hang if a source fails at the same moment (#2969). Default engine, present in
    v0.19.0. If a source errors just as Conduit is stopping, shutdown can block indefinitely. No data
    is lost: kill the process and the unacked records are replayed on restart. Fix planned for
    v0.20.1.
  • Postgres CDC can lose a row on restart when transactions interleave
    (conduit-connector-postgres#331).     Present since 2022, including v0.19.0. Trigger: a transaction
    that started before the last checkpoint commits after it while other transactions commit in
    between (for example a long-running transaction alongside short ones), and then Conduit
    restarts. On resume, that transaction's changes are skipped and never delivered. Restarts with
    no long-running transaction in flight are not affected. Fixed in Postgres connector v0.14.3
    (released 2026-10-08), which becomes the built-in connector in v0.20.1.
  • Postgres CDC can lose rows loaded in one statement (for example COPY or a multi-row
    insert) if Conduit restarts partway through them.     Present in v0.19.0 and earlier. Postgres
    gives all rows from such a statement the same log position, so after a restart between two of
    them the rest are skipped, and the connector can confirm progress past rows it has not
    delivered yet. Fixed in the same v0.20.1 connector patch.
  • Avro decoder advisories GO-2026-5046, GO-2026-5047, GO-2026-5048 (CPU exhaustion, integer
    overflow, unbounded map allocation in hamba/avro, archived upstream with no fix). Reachable when
    Conduit decodes untrusted Avro through the built-in Avro processor, or through the built-in
    Postgres connector's schema extraction. v0.20.1 partly mitigates them (conduit-commons v0.7.0 caps Avro arrays and maps at
    1,000,000 elements). v0.21 clears them by moving to a maintained fork.
  • #2929 (arch-v2 preview engine): a persistent connector-state write failure stalls without
    a status signal.     No data is lost: upstream acks are withheld (#2925). But the pipeline keeps
    reporting healthy, logs one error per flush, and the upstream keeps retention, for example a
    growing Postgres replication slot. Transient failures recover on their own. Very unlikely on the
    default store for single-source pipelines such as the RAG template. Watch replication-slot lag.
  • #2898 (default engine): a failed Running status write can orphan a live run. Trigger: a
    transient pipeline-store write failure on start. Impact: a run that can't be stopped or drained;
    starting it again puts two readers on one source (duplicates, positions can move backwards).
  • #2899 (both engines): a failed status write during recovery can leave a live run Degraded.
    Trigger: a status-store write failure during recovery. Impact: the run keeps going, can't be
    stopped, can be started twice, and can trip exit-on-degraded.
  • #2900 (default engine): source teardown waits its full 10s after a plugin stream error.
    Trigger: a source plugin's stream ends with an error. Impact: about 10s of extra delay before
    Degraded/Recovering and on shutdown. No data loss.
  • #2913: sending a redacted settings map back stores "***" as the value. Trigger: an API
    update that echoes settings read from the API. Impact: the real secret is overwritten. Send real
    values.
  • #2908: generate.provider in conduit.yaml is not read. Use --provider.
  • #2907: processor-plugins install / uninstall --help list the wrong exit codes.
  • postgres-s3 template is not verified end to end. The built-in S3 connector (v0.9.3) fails
    against MinIO and other S3-compatible stores; the fix needs an S3 connector release with
    conduit-connector-s3#994 and reaches Conduit in v0.21. The template targets AWS S3.

Deferred

  • A single pipeline engine. Instead of switching the default to arch-v2, v0.21 designs and
    prototypes one engine that keeps arch-v2's batching and fan-out and runs pipeline stages
    concurrently, as the default engine does. The target is to make it the default in v0.22, after
    it matches or beats the current default on throughput and passes the chaos suite.
  • Debezium-parity Postgres CDC → v0.21. The built-in Postgres connector stays at v0.14.2 until
    then.
  • Protobuf decode (Confluent wire format) → v0.21.
  • → v0.22: embedded Conduit with external connectors (embed Slice 2), a Node client, a Qdrant
    destination, and a source-side DLQ path.
  • Python connector SDK and conduit-client GA ship on their own schedule. Neither blocks this
    release.

Verify this release

Requires cosign v3 or later.
Download checksums.txt, checksums.txt.sigstore.json and the files you want into one directory,
then:

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.0' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json \
  checksums.txt

# only once the signature verifies (on macOS: shasum -a 256 --check --ignore-missing)
sha256sum --check --ignore-missing checksums.txt

The Docker image:

cosign verify \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.0' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/conduitio/conduit:v0.20.0

cosign verify resolves the tag to a digest and checks the signature on that digest. To pin what
you run, use the digest it prints: ghcr.io/conduitio/conduit@sha256:....

Full changelog: v0.19.0...v0.20.0

Commit list

Changelog

Release Artifacts

OSX Darwin

Linux

Windows

Verifying

checksums.txt is signed with Sigstore keyless signing (checksums.txt.sigstore.json), and each archive has an SPDX SBOM. The Docker image is signed by digest. See Verifying a release.

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.0' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json checksums.txt
sha256sum --check --ignore-missing checksums.txt

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.20.0

conduit-release-view

Contributors

dependabot, devarismeroxa, and Guflly
Assets 24
Loading