Release v0.21.0 · ConduitIO/conduit

Warning

Upgrade to v0.21.2. v0.21.0 refuses to start every pipeline that has no dead-letter-queue block, including the hello-pipeline example and the template gallery, with pipeline.dlq_not_recoverable. The default DLQ never receives a record (the first nack stops the pipeline and the record is not acked), so v0.21.2 refuses a hash-only log DLQ only when it does receive records. No data is lost by the refusal: the pipeline does not start. v0.21.1 was tagged but never built; skip it.

Changelog

5d06327: Go tools: bump github.com/klauspost/compress in /tools (#3232) (@dependabot[bot])
b9883e5: Merge dialguard/c2 (conduit-GHSA-m3mw-fpw8-w765): connector dial guard via conduit-commons dialer seam (@devarismeroxa)
31bb927: Merge fix/ghsa-gwvc-plugin-env (conduit-GHSA-gwvc-9h8p-x5r7): standalone plugin environment isolation (@devarismeroxa)
08028a6: Merge fix/ghsa-m3mw-c2-hardening (conduit-GHSA-m3mw-fpw8-w765): settings/egress hardening, DLQ guard (@devarismeroxa)
ab570fd: Merge ghsa/api-hardening (conduit-GHSA-v79j-j88p-m4ff): API token auth, CSRF, security headers (@devarismeroxa)
2c3a405: Merge main (3d0dfe5) into ghsa/api-hardening (@devarispbrown)
90b1dba: bench: v0.21 CDC parity kit and results (#3228) (@devarismeroxa)
d62be08: bench: v0.21 arch-v2 parity results (#3247) (@devarismeroxa)
461b2c4: build(deps): bump the v0.21 module cascade (#3051) (@devarismeroxa)
263d038: build: build release binaries with -trimpath (#3252) (@devarismeroxa)
9e6be91: chore(benchi): bump CDC parity loader to pgx v5.9.2 (#3245) (@devarismeroxa)
bd58adc: chore(certification): certify the migrate connectors on the v0.21 swept releases (#3202) (@devarismeroxa)
9559d07: chore(ci): add make check hygiene gate and scope module resolution to ConduitIO (#3211) (@devarismeroxa)
37d196e: chore(deploy): pin examples to conduit v0.20.2 (#3113) (@github-actions[bot])
3f76c92: chore(deploy): pin examples to conduit v0.20.4 (#3171) (@github-actions[bot])
1df6033: chore(deps): bump built-in kafka connector to v0.14.2 and generator to v0.10.8 (#3244) (@devarismeroxa)
69bc915: chore(deps): bump built-in kafka connector to v0.15.0 (#3258) (@devarismeroxa)
3de00c3: chore(deps): bump built-in log to v0.8.0 and file to v0.11.0 (@devarismeroxa)
6d6f6db: chore(deps): bump built-in postgres connector to v0.17.5 (#3277) (@devarismeroxa)
03051c8: chore(deps): bump conduit-commons to v0.12.1 (#3257) (@devarismeroxa)
9709906: chore(deps): point postgres/kafka/s3 at the fixed dial-guard connector heads (@devarispbrown)
f18d926: chore: bump built-in kafka + s3 connectors (swept perf releases) (#3209) (@devarismeroxa)
3454bb4: chore: bump built-in postgres to v0.17.1 (source default batch 1000) (#3213) (@devarismeroxa)
844e8e3: ci(bench): non-blocking engine benchmark comment on engine PRs (#3030) (@devarismeroxa)
2d453f1: ci(chaos): count standalone processor runtime as data path (#3050) (@devarismeroxa)
c29b293: ci(kcmigrate): nightly upstream key-drift check for the rule snapshots (#3120) (@devarismeroxa)
0e3f1f0: ci(lint): forbid the Avro codec's package-level API (#3090) (@devarismeroxa)
067beb9: ci(test): run memory-heavy test packages serially to stop runner OOM kills (#3224) (@devarismeroxa)
b38d0c0: ci(tier1-review): post the failing status with its description (#3276) (@devarismeroxa)
8f7b694: ci: add engine coverage ratchet gate (#3053) (@devarismeroxa)
b1da04a: ci: add release-branch strategy (release/v0.20.x CI, scheduled dispatcher, docs) (#2999) (@devarismeroxa)
c545ab0: ci: bump the deploy examples' image pin after each stable release (#3061) (@devarismeroxa)
d1b8407: ci: connector release gate for merged PRs with an open finding (stacked on #3266) (#3268) (@devarismeroxa)
8df6ce2: ci: fail a PR that lowers a go.mod version unless its body says why (#3265) (@devarismeroxa)
010630b: ci: fail a patch release whose notes are only the generated changelog (stacked on #3267) (#3269) (@devarismeroxa)
0572eb5: ci: fail a release tag that does not contain the fixes it was meant to carry (#3267) (@devarismeroxa)
d95e300: ci: make DeVaris the sole code owner of Tier 1 paths and the gates (#3259) (@devarismeroxa)
ae6c6a6: ci: make golangci-lint results independent of cache state (#3077) (@devarismeroxa)
977b6bb: ci: make the benchi regression gate live (>10% slower than base fails) (#3187) (@devarismeroxa)
55aa441: ci: stop the image pin bump from sending two Authorization headers (#3109) (@devarismeroxa)
fd3b558: ci: tier1-review status from the PR's declared tier and review comments (#3266) (@devarismeroxa)
9f11b46: deps: bump built-in postgres connector to v0.17.2 (#3217) (@devarismeroxa)
6248446: deps: bump built-in postgres to v0.14.4 and s3 to v0.9.5 (#3064) (@devarismeroxa)
efa881d: deps: bump built-in postgres to v0.15.0 and s3 to v0.10.0 (#3173) (@devarismeroxa)
d76f660: deps: bump built-in s3 connector to v0.11.2 (#3220) (@devarismeroxa)
bb588ef: deps: bump conduit-commons to v0.12.0 (#3223) (@devarismeroxa)
79f91b6: deps: pin dial-guard connector deps to released tags (@devarispbrown)
fa441bf: docs(adr): propose lossless structured values via protocol v3 (#3100) (#3132) (@devarismeroxa)
c2ee14f: docs(arch): single pipeline engine — staged batch engine design + ADR (#2971) (@devarismeroxa)
22d5e01: docs(architecture): engine guarantees, life of a record, engine contributing guide (#3031) (@devarismeroxa)
722dab4: docs(catalog): public connector tier + license scorecard (#3186) (@devarismeroxa)
4a51030: docs(catalog): record the mysql and elasticsearch transfer to ConduitIO (#3042) (@devarismeroxa)
44add69: docs(catalog): v0.21 certification gaps for Kafka Connect migration targets (#3034) (@devarismeroxa)
b913b24: docs(design): Debezium-compatible output for Postgres CDC (#3033) (@devarismeroxa)
2f262e3: docs(design): Kafka mirror passthrough for headers, partition and timestamp (#3038) (@devarismeroxa)
03f4250: docs(design): MySQL source type semantics for NULL, DECIMAL and unsigned BIGINT (#3040) (@devarismeroxa)
5ea34ab: docs(design): S3 destination parity with the Kafka Connect S3 sink (#3083) (@devarismeroxa)
bbf23f6: docs(design): SDK parity automation and the open plugin protocol (#2948) (@devarismeroxa)
b8f36f3: docs(design): acceptance suite v2 and what the registry records for Certified (#3036) (@devarismeroxa)
db9e102: docs(design): exact resume for the S3 source (#3099) (@devarismeroxa)
a015b31: docs(design): kafka is not yet a Certified migrate target (#3060) (@devarismeroxa)
ff6be4e: docs(design): migrate kafka-connect mapping tables, report contract, harness corpus, MCP parity (#3057) (@devarismeroxa)
6709305: docs(design): migrate kafka-connect surface for Debezium Postgres offset import (#3029) (@devarismeroxa)
1e77e88: docs(design): migrate kafka-connect v1, kc lint, kc diff (#3026) (@devarismeroxa)
15e35e3: docs(design): operator control over ambient credentials in built-in connectors (#3041) (@devarismeroxa)
0de7657: docs(design): processor input isolation for DLQ fidelity (#3153) (#3156) (@devarismeroxa)
5ff4ae1: docs(design): record sign-off decisions for Protobuf decode (#2882) (@devarismeroxa)
3605ed5: docs(design): refuse to round wide integers at the plugin boundary (#3100) (#3126) (@devarismeroxa)
b966dd0: docs(design): secrets at rest, store ${VAR} references not values (#2988) (#3055) (@devarismeroxa)
26e5d6b: docs(design): secrets-at-rest, Python GA, and processor batch API (#3178) (@devarismeroxa)
0b630ee: docs(engine): guarantees page and life-of-a-record guide (#3194) (@devarismeroxa)
b96f813: docs(llms): add acceptance suite v2 to llms.txt (#3082) (@devarismeroxa)
5f51ae2: docs(postmortem): Avro encoding used another pipeline's named types (#3096) (@devarismeroxa)
d178fab: docs(postmortem): Record.Clone shared nested data between copies (#3150) (@devarismeroxa)
5811e3c: docs(postmortems): MySQL CDC loses rows on the second restart inside a binlog event (#3095) (@devarismeroxa)
6ffaf60: docs(release): draft v0.21.0 release notes; mark shipped roadmap items (#3193) (@devarismeroxa)
3438aaa: docs(release): finalize v0.21.0 release notes and release-fixes gate (@devarismeroxa)
b3cb069: docs(releases): wrap a line that breaks MD013 (#3275) (@devarismeroxa)
ad532ee: docs(roadmap): link v0.22 items to their PRs (#3215) (@devarismeroxa)
1a56f0e: docs(roadmap): pull TypeScript connectors forward to v0.25 (full tier, parallel with Java) (#3196) (@devarismeroxa)
e2df27c: feat(api)!: bind the API to loopback by default and require a token off loopback (@devarispbrown)
c7bcb96: feat(api): add a bearer-token gate for the HTTP, gRPC and inspector APIs (@devarispbrown)
93ed454: feat(api): count rejections per code, rate-limit the WARN, accept the token as a WebSocket subprotocol (@devarispbrown)
ff8e955: feat(cli): migrate kafka-connect skeleton with --json and exit codes (#3074) (@devarismeroxa)
fca6907: feat(connector): refuse standalone connectors while connectors.egress.allow is set (@devarispbrown)
0e6d3de: feat(connectors): operator policy for ambient credentials in built-in connectors (#3108) (@devarismeroxa)
b82845d: feat(deploy): add docker-compose quickstart and deploy examples (#3056) (@devarismeroxa)
a369dc7: feat(docker)!: run as non-root, listen on all interfaces with a required token, document hardening (@devarispbrown)
c73fb0c: feat(funnel): destination-write batching (arch-v2) + batch ownership fix (#3201) (@devarismeroxa)
e19c8e2: feat(kcmigrate): Debezium Postgres rules and goldens (#3032 follow-up) (#3188) (@devarismeroxa)
2d755ca: feat(kcmigrate): Kafka Connect config parsers with fuzzing (#3070) (@devarismeroxa)
bd9775e: feat(kcmigrate): MirrorMaker 2 (MirrorSourceConnector) rules (#3118) (@devarismeroxa)
5d3407a: feat(kcmigrate): MirrorMaker checkpoint and heartbeat connectors (no-pipeline classes) (#3119) (@devarismeroxa)
ff8fe4a: feat(kcmigrate): S3 sink rules and goldens (#3003) (#3181) (@devarismeroxa)
ff9f040: feat(kcmigrate): absent-key defaults and upstream key snapshots (#3081) (@devarismeroxa)
7f9277b: feat(kcmigrate): block SASL_PLAINTEXT instead of translating it (#3255) (@devarismeroxa)
b20eb92: feat(kcmigrate): compatibility report types and JSON schema (#3069) (@devarismeroxa)
2d10f12: feat(kcmigrate): config layers, required settings, class notes and finish actions (#3117) (@devarismeroxa)
fd3c324: feat(kcmigrate): emit pipeline YAML, secrets template and pipeline validation (#3078) (@devarismeroxa)
01e2aee: feat(kcmigrate): rule data format and classification engine (#3073) (@devarismeroxa)
e574369: feat(kcmigrate): shared and worker rules for Kafka client, error and converter keys (#3080) (@devarismeroxa)
62ebf51: feat(mcp): kc_migrate tool mirroring migrate kafka-connect (#3195) (@devarismeroxa)
0e5c6eb: feat(metrics): per-connector record and error counters (#3013 slice) (#3185) (@devarismeroxa)
9509a47: feat(plugin): warn when sensitive environment variables are passed to standalone plugins (#3045) (@devarismeroxa)
5ffbd7f: feat(processor): add debezium.encode built-in processor (#3032) (#3183) (@devarismeroxa)
a83e2a7: feat(processor): built-in protobuf.decode (PB-5) (#3179) (@devarismeroxa)
126d777: feat(provisioning): store ${VAR} references, resolve at provision (secrets-at-rest, #3018) (#3180) (@devarismeroxa)
b59a2a2: feat(registry): optional license and tier on index entries (#2963) (@devarismeroxa)
1f0f1a1: feat(scaffold): add conduit connector new --lang python (#3182) (@devarismeroxa)
004918d: feat(schemaregistry): map protobuf schemas to schema.TypeProtobuf (#3176) (@devarismeroxa)
ddca592: feat(ui): prompt for the API token, keep it in sessionStorage, send it as Bearer (@devarispbrown)
5646100: feat(validate): flag sdk.batch.size > 1 under arch-v2 (#3174) (@devarismeroxa)
8a4068e: feat: hand the connector dial guard to built-in connectors through the dialer seam (@devarispbrown)
f741ad0: feat: negotiate connector protocol v3 for lossless structured values (#3191) (@devarismeroxa)
7003537: feat: send opencdc.v2 records to processors declaring structured_values_v2 (#3192) (@devarismeroxa)
9ac7870: fix(api): bind the connector utilities server to loopback (@devarispbrown)
4e9b4c9: fix(api): do not treat an Origin with userinfo, path or query as same-origin (@devarispbrown)
0813688: fix(api): hold the run tomb open during setup, revoke connector tokens on failed start, document the container Host-check gap (@devarispbrown)
a35f4e4: fix(api): minimise /readyz, add UI security headers, harden token and Origin/WebSocket checks (@devarispbrown)
79dc3f0: fix(api): refuse cross-origin, rebound-Host and non-JSON mutating requests (@devarispbrown)
ae1f302: fix(cli): --json renders coded errors as JSON for client commands (#3136) (@devarismeroxa)
f16d208: fix(cli): document all bundle exit codes in --help (#3163) (@devarismeroxa)
3dfaf35: fix(cli): render usage errors on envelope commands and exit 2 (#3144) (@devarismeroxa)
f3da596: fix(cli): stable error code and location for a malformed conduit.yaml (#3162) (@devarismeroxa)
5ddb3c6: fix(conduit): make Handle.Stop deterministic for an expired context (#3274) (@devarismeroxa)
1700317: fix(connector): make Source.Read honour its context (#2980) (@RishavRaj20)
1620bb1: fix(connector): restore the deleted instance on a failed delete rollback (#3137) (#3241) (@devarismeroxa)
44145ed: fix(connector): stamp readAt and source connector ID on records with nil metadata (#3133) (@devarismeroxa)
3d0dfe5: fix(deps): bump Go toolchain to 1.26.10 for GO-2026-6809 (#3250) (@devarismeroxa)
0c1fff9: fix(embed): stop process-wide loggers racing across embedded runtimes (#3165) (@devarismeroxa)
4752f0f: fix(grpcutil): do not log inspector messages from the websocket proxy (#3249) (@devarismeroxa)
e397765: fix(json.decode): decode integers losslessly, no rounding above 2^53 (#3134) (#3189) (@devarismeroxa)
4c7678f: fix(kcmigrate): carry Kafka Connect's consumer group and offset reset into migrated sinks (#3236) (@devarismeroxa)
df5cd27: fix(kcmigrate): generated variables avoid names that ${env:...} references use (#3115) (@devarismeroxa)
af43231: fix(kcmigrate): source.cluster.alias is required by MirrorSourceConnector (#3121) (@devarismeroxa)
52acf2a: fix(lifecycle): WaitPipeline waits for the run's goroutines after cleanup (#3087) (#3240) (@devarismeroxa)
2b3a473: fix(lifecycle-poc): report the first source teardown error from Worker.Close (#3145) (@devarismeroxa)
1b7290d: fix(log): log record positions as a salted hash, never as is (#3253) (@devarismeroxa)
d5ca728: fix(pipeline): guard Service's instance maps with a mutex (#3261) (@devarismeroxa)
7bb11b4: fix(pipeline): make a pipeline config write's error mean what it says (#3140) (@devarismeroxa)
a88250f: fix(plugin): bump conduit-connector-sdk to fix source Run tomb.Go race (#3208) (@devarismeroxa)
79c622d: fix(plugin): require a connector token on every call to the connector utilities server (@devarispbrown)
c7e22a0: fix(plugin): restore the connector SDK tomb.Go fix that #3192 reverted (#3229) (@devarismeroxa)
10e3f1e: fix(plugin): say a future release restricts the plugin environment (#3065) (@devarismeroxa)
adba553: fix(plugin): scope built-in plugin schema services to their runtime (#3104) (@devarismeroxa)
1af5ca7: fix(plugin): widen standalone env warning coverage and make its tests hermetic (#3054) (@devarismeroxa)
e62826f: fix(processor): bump conduit-processor-sdk to v0.8.0 for schema fidelity (#3085) (@devarismeroxa)
ceb890f: fix(processor): do not repeat record values in JavaScript exceptions (#3260) (@devarismeroxa)
0c9c510: fix(processor): do not repeat record values in library errors, redact the schema registry URL (#3256) (@devarismeroxa)
ccbb824: fix(processor): do not repeat the request URL or response body in webhook and ollama errors (#3254) (@devarismeroxa)
6b1a741: fix(processor): isolate processor input for DLQ fidelity (#3153) (#3190) (@devarismeroxa)
d7a023b: fix(processor): keep record keys and request bodies out of trace logs (#3248) (@devarismeroxa)
d5b5ddb: fix(processor): log the type of a Cohere retry error, not its text (#3270) (@devarismeroxa)
e40d0ab: fix(processor): reject cyclic or oversized custom.javascript output instead of crashing (#3226) (@devarismeroxa)
abdca27: fix(registry): do not echo URL user info in download and fetch errors (#3251) (@devarismeroxa)
2fa9a90: fix(schemaregistry): key the client cache on SHA-256 over type, references and text (#3105) (@devarismeroxa)
aeab728: fix(split): don't panic on nested fields, never discard the Resolve error (#3154) (@devarismeroxa)
81c6306: fix(standalone): never let an abandoned WASM command wedge the processor; bound install validation (#2996) (@devarismeroxa)
656349a: fix(test): drain the persister before closing the DB in provisioning tests (#3164) (@devarismeroxa)
f151340: fix(test): shrink the concurrent-processors chaos test to fit CI (#3166) (@devarismeroxa)
c24babf: fix(valuecorrectness): reap runtime goroutine and fix v2 shutdown deadlock (#3200) (@devarismeroxa)
67ad84d: fix: apply file path confinement to connector validation requests (@devarispbrown)
68e7c12: fix: close the remaining settings-boundary gaps (lifecycle events, other file settings, open-time checks) (@devarispbrown)
c2a03be: fix: connector egress fails closed; dial-time guard; S3 endpoint is not guessed (@devarispbrown)
417bdd8: fix: connector egress policy, run defaults, postgres ambient files, deletion, NAT64 (@devarispbrown)
fefc0ba: fix: deny pipeline settings ambient access to env, network and files (@devarispbrown)
5679267: fix: exit promptly on SIGTERM with SDK destination batching (#3102) (@devarismeroxa)
59a22ae: fix: pass SSL_CERT_FILE, SSL_CERT_DIR and the proxy variables to standalone plugins (@devarispbrown)
f3e76d7: fix: redact expanded env values from errors, refuse unresolvable deletes, $$ escape, hardening follow-ups (@devarispbrown)
b911637: fix: refuse to start a pipeline whose DLQ would drop record content (@devarispbrown)
4abc196: fix: standalone connector plugins no longer inherit the engine environment (@devarispbrown)
5443a1d: hardening: standalone plugin environment allowlist (#3043) (@devarismeroxa)
0e1a94c: lint: constants for the proxy variable names (@devarispbrown)
3c145a4: lint: derive lower-case proxy names (@devarispbrown)
2c0447b: perf(builtin): stop cloning records at the source→engine boundary (#3198) (@devarismeroxa)
87a49d7: perf(funnel): default destination write batch to 3000 rows (#3203) (@devarismeroxa)
58351fa: perf(funnel): pipeline destination flush with source read (read-ahead) (#3210) (@devarismeroxa)
193446f: perf: bump built-in file + postgres connectors (#3199) (@devarismeroxa)
0f43773: refactor(api): drop the unreachable TLS Origin branch, report token-file path, fix comments (@devarispbrown)
cedc8ce: refactor(api): print coded startup errors, address lint findings (@devarispbrown)
3c85161: style(api): gofmt browser_test.go (@devarispbrown)
f109d0b: test(api): do not fail when the server rejects a reflection stream before Send (@devarispbrown)
afb5a0d: test(api): read counters without prometheus/testutil (no new indirect dependency) (@devarispbrown)
ac9efeb: test(chaos): SIGTERM while a failed source is still draining (#3143) (@devarismeroxa)
58dce5f: test(chaos): fan-out deep-clone case across SIGKILL (#3243) (@devarismeroxa)
71ee7d8: test(chaos): run the pipelined-flusher crash restart unbatched (#3230) (@devarismeroxa)
73a3431: test(config): use a named constant for the padded batch key (#3177) (@devarismeroxa)
3141349: test(connector): stop asserting Wait orders the stop goroutine's return (#3125) (@devarismeroxa)
de14091: test(debezium): golden captures from real Debezium Postgres 2.7.4 and 3.7.0 (#3075) (@devarismeroxa)
f3f2492: test(dev): wait for the restart mode log line, not just the second start (#3110) (@devarismeroxa)
74e4802: test(funnel): make WatchConnectorErrors reader-lifetime tests deterministic (#3068) (@devarismeroxa)
b63c452: test(lifecycle): make bench-engine, in-process reference pipeline shapes (#3028) (@devarismeroxa)
6ea649e: test(lifecycle): simulated-connector test kit for both pipeline engines (#3027) (@devarismeroxa)
ae74c3c: test(lifecycle): wait for runs to exit before asserting none is alive (#3141) (@devarismeroxa)
49ccf40: test(migrate): Kafka Connect corpus and integrity checks for the migration harness (#3071) (@devarismeroxa)
4dbbc2b: test(migrate): harness runner for report contract, completeness and MCP parity; CI job (#3072) (@devarismeroxa)
644b61a: test(migrate): in-process migrator so every ready corpus case runs in the test job (#3116) (@devarismeroxa)
d3d0bf0: test(migrate): scope the MM2 topic exemption to comments; fail on unknown rule classes (#3098) (@devarismeroxa)
bc47d55: test(processor): adopt rapid; property tests for built-in transforms (#3139) (@devarismeroxa)
8c3c4d3: test(processor): nack only the record that failed schema decode (#2968) (#3142) (@devarismeroxa)
a1dce06: test(registry): freshness-only re-sign cannot change tier or license (#3063) (@devarismeroxa)
b2cba77: test(stream): make the fan-out deep-clone test deterministic (#3246) (@devarismeroxa)
17a4244: test(templates): poll until every seeded row lands in S3 (#3112) (@devarismeroxa)
8cb6c8a: test(valuecorrectness): check bytes and values across pipelines that share Avro type names (#3091) (@devarismeroxa)
519701c: wip: dial guard via conduit-commons dialer seam (engine side, e2e, docs) (@devarispbrown)

Release Artifacts

OSX Darwin

Linux

Windows

📦 conduit_0.21.0_Windows_arm64.zip
📦 conduit_0.21.0_Windows_x86_64.zip

Verifying

checksums.txt is signed with Sigstore keyless signing (checksums.txt.sigstore.json), and each archive has an SPDX SBOM. The Docker image is signed by digest. See Verifying a release.

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.21.0' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json checksums.txt
sha256sum --check --ignore-missing checksums.txt

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.21.0

Navigation Menu

Repository navigation

v0.21.0

Choose a tag to compare

Sorry, something went wrong.