Skip to content

v0.20.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 15 Oct 08:44
· 163 commits to main since this release
v0.20.1
49258fe
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Conduit v0.20.1

v0.20.1 is a patch release. It fixes data-loss bugs in the built-in Postgres, S3 and Kafka
connectors, and lifecycle bugs where a failed status write could leave a run that could not be
stopped, or let one pipeline run twice. Embedded connectors: postgres v0.14.2 → v0.14.4, s3 v0.9.3 →
v0.9.5, kafka v0.12.5 → v0.12.6. Read Before you upgrade first: some Postgres
pipelines stop on purpose, the first restart can re-deliver records, and downgrading is not safe for
every pipeline.

Upgrading from v0.19.0? Read the v0.20.0 notes
first. Their behavior changes still apply.

Upgrade if you run v0.20.0

  • Postgres CDC could lose rows on restart (conduit-connector-postgres#331), since 2022:
    interleaved transactions and rows sharing one log position (COPY, multi-row inserts). Fixed in
    v0.14.3 (#2938), included in v0.14.4 (#3052).
  • Kafka destination could ack a record that failed to write in a multi-record batch
    (conduit-connector-kafka#308). Fixed in v0.12.6 (#3049).
  • Postgres destination acked a batch whose COMMIT failed (conduit-connector-postgres#342), since
    v0.6.0. Fixed (conduit-connector-postgres#351).
  • Postgres DELETE stopped Avro pipelines (conduit-connector-postgres#379), since v0.11.0: the
    pipeline restarted on the same record forever while the slot kept WAL. Fixed
    (conduit-connector-postgres#385); upgrading alone unsticks it.
  • Postgres wrote unchanged TOASTed columns as NULL (conduit-connector-postgres#317). Fixed
    (conduit-connector-postgres#387); one case still halts, see Known issues.
  • Postgres NUMERIC (conduit-connector-postgres#364,
    conduit-connector-postgres#368, conduit-connector-postgres#375): stale scale at the destination;
    source panic on NaN/Infinity; source halt on an unconstrained numeric of 10000 or more. Fixed (conduit-connector-postgres#366,
    conduit-connector-postgres#372, conduit-connector-postgres#382).
  • Postgres lifecycle (conduit-connector-postgres#346,
    conduit-connector-postgres#347, conduit-connector-postgres#354): wedged CDC, panic on stop
    during start, Teardown under a running Write, a goroutine leak. Fixed
    (conduit-connector-postgres#353, conduit-connector-postgres#357,
    conduit-connector-postgres#363).
  • S3 CDC skipped objects for good (conduit-connector-s3#1029, conduit-connector-s3#1032). Fixed
    in v0.9.5 (conduit-connector-s3#1037).
  • S3 destination panicked on a delete record, and on every restart after
    (conduit-connector-s3#1030). Fixed (conduit-connector-s3#1039).
  • S3-compatible stores such as MinIO now work via aws.endpoint and aws.pathStyle, both off by
    default (#2940).
  • Shutdown could hang if a source failed during stop (#2969). Fixed (#2975).
  • A failed status write could orphan a live run, degrade a running pipeline, or start a second run
    on one source     (#2898, #2899). Fixed (#2941, #2942, #2955, #2962).
  • One failed connector state write failed every connector in its shared write (#2930), including
    other pipelines'. Fixed (#2950).

Before you upgrade

  • Postgres: some pipelines stop on purpose (v0.14.4).
    • REPLICA IDENTITY USING INDEX on an index whose key columns exclude the primary key is refused at
      start with postgres.replica_identity.excludes_key. Such tables emitted deletes with a null key,
      which the destination applied as a no-op, so deleted rows stayed. Use REPLICA IDENTITY FULL or
      a unique index that includes the primary key, and check destinations for rows that should be gone.
    • Under Avro, infinity/-infinity in a date, timestamp or timestamptz halts with
      postgres.source.temporal_not_representable. It used to be written silently as 1970-01-02 /
      1969-12-31, so those rows are wrong downstream. Restart once with
      logrepl.withAvroSchema: false from the same position, then switch back. NaN in a
      numeric(p,s) halts with postgres.source.numeric_not_representable (it used to panic).
  • Postgres: record shape changes (v0.14.4).
    • Under the default replica identity, deletes and key-changing updates no longer carry
      Payload.Before; the key is in Key. Unchanged TOASTed columns are omitted instead of NULL
      (with Avro on, the source halts instead, conduit-connector-postgres#383). Under REPLICA IDENTITY FULL they carry their value.
    • An unconstrained numeric is now an Avro string. Consumers see a new schema version and must
      handle "NaN", "Infinity" and "-Infinity".
    • The destination no longer checks that a column is numeric: a decimal string is stored in any
      column whose text input accepts it. That is a silent coercion if a schema maps a numeric field to
      the wrong column. Rows written by the old rounding (2.50 stored as 3) are not repaired.
  • Postgres: duplicates on the first restart. v0.14.2 positions carry no transaction information,
    so the first restart after the upgrade uses a fallback. It loses nothing but can re-deliver acked
    records: up to everything committed while the checkpointed record's transaction was open, plus
    that transaction's delivered rows. That is bounded by how long the transaction was open, not by a
    small count. Non-idempotent sinks and processors see the duplicates once; upsert-by-key
    destinations absorb them. After the first new checkpoint, resumes are exact. Positions are now
    format version 2 (tx_commit_lsn, tx_seq).
  • Postgres: downgrading is not safe. Downgrading to v0.20.0 can lose CDC rows on the first restart
    and may stop the subscription; the log then shows handler error: out of order commit while the
    pipeline still reports running. Do not downgrade a pipeline with interleaved transactions or COPY
    traffic in flight. If you must roll back, expect to re-snapshot.
  • Postgres: SCRAM channel binding is off by default. v0.14.4 maps channel_binding=prefer (the
    default, or set in the URL or PGCHANNELBINDING) to disable in source and destination. Servers with an Ed25519 certificate,
    and TLS re-terminating proxies that pass SASL through, failed with SCRAM channel binding negotiation error or channel binding check failed; they now work. To keep channel binding, set
    channel_binding=require in the connection URL.
  • S3: CDC re-delivers recent objects (v0.9.5). After a restart and at the snapshot handoff,
    objects modified within lookbackPeriod (default 5 minutes) of the newest one, plus everything
    written during the snapshot, are delivered again. Destinations must tolerate duplicates. A freshly
    bulk-loaded bucket delivers its last window twice. If tracking state would exceed about 256 MiB
    (roughly a million objects), the source shortens its window and logs
    s3.source.dedupe_window_shrunk; lower lookbackPeriod for such pipelines. It halts with
    s3.source.dedupe_window_overflow when too many versions share one second, or when future-stamped
    entries (a store clock ahead of the host) do not fit the budget. See the
    s3 v0.9.5 notes.
  • S3: writes that got HTTP 200 with an <Error> body now fail and are retried. Before, they were
    acked. Only stores that answer that way are affected. To roll back to v0.20.0, first remove
    aws.endpoint and aws.pathStyle from every pipeline: S3 v0.9.3 refuses them as unrecognized
    parameters. Positions are unchanged between v0.9.3 and v0.9.4 (#2940).
  • Settings updates with "***" (#2926). A redacted "***" keeps the stored secret only if
    nothing else in the update changes; otherwise the update is refused. Clients that change one field
    and send the rest back as "***" must send the real secret values.
  • Avro decode limit. An array or map that declares more than schema.avro.max-elements elements
    fails to decode with schema.avro.limit_exceeded; nothing is truncated. Default 1,000,000; 0
    means no limit; a negative value fails config validation. Set it with
    --schema.avro.max-elements, CONDUIT_SCHEMA_AVRO_MAX_ELEMENTS or schema.avro.max-elements in
    conduit.yaml. If you decode legitimate arrays or maps over 1,000,000 elements, raise it before
    you upgrade.     It covers decodes inside the Conduit process, not standalone plugins. Sources still
    encode records over the limit, so a destination can reject its own source's record, failing the
    whole batch. With the default DLQ the pipeline stops Degraded or, with max-retries: -1, retries
    forever in Recovering (#2997). Runbook: docs/operations/avro-limit-exceeded.md. The codec is
    now iskorotkov/avro/v2, a maintained fork; the wire format is unchanged.
  • Storage engines. badger v4.9.1 → v4.9.6, modernc.org/sqlite v1.38.0 → v1.46.1. New CI tests
    open databases across old and new versions in both directions and resume a stored position each
    way. badger lists no on-disk format change.
  • New error codes: pipeline.stopping, pipeline.status_persist_failed,
    pipeline.config_changing, pipeline.archv2_destination_batching_unsupported, three
    orchestrator.redacted_setting_* codes, schema.avro.limit_exceeded and
    registry.processor_validation_timeout.

Behavior changes

Change v0.20.0 v0.20.1 Code PR
"***" in a connector, processor or DLQ settings update stored literally, or failed validation kept only if nothing else changes; otherwise refused orchestrator.redacted_setting_with_other_changes (and _without_stored_value, _plugin_changed), InvalidArgument #2926
arch-v2: destination or DLQ with sdk.batch.size > 1 could stall while reporting running refused at start and on live apply pipeline.archv2_destination_batching_unsupported, FailedPrecondition #2987
Start during a config write started on a changing config refused, retry pipeline.config_changing, Unavailable #2991
Avro array or map over the limit decoded decode fails schema.avro.limit_exceeded, ResourceExhausted #2931
One connector's state write fails in a shared write all connectors in it failed only that one fails connector.state_persist_failed #2950
arch-v2: connector state write or deferred ack fails kept reporting running run goes Recovering, restarts from last stored position — #2944
S3 HTTP 200 with <Error> body acked failed, retried — #2940
Start while the previous run is finishing granted refused, retry pipeline.stopping, Unavailable #2955
Config change while a run is starting or in recovery backoff allowed unless Running refused pipeline.running #2955, #2962
Stop racing a source failure (default engine) hung returns PubNode has stopped running — #2975
Postgres SCRAM over TLS, channel_binding unset or prefer binding used if offered not used — #3052
Standalone WASM processor install check times out 5s, could hang 30s, honours caller deadline registry.processor_validation_timeout, exit 3 #3039
conduit generate provider conduit.yaml ignored --provider > env > conduit.yaml > auto-detect — #2923

Notes:

  • "***": the three codes are HTTP 400, CLI exit 2. Create and apply calls refuse "***", and a
    literal *** can only be set through YAML. Keys left out of an update are still removed.
  • arch-v2 batching: remove sdk.batch.size and sdk.batch.delay from the destination, or use
    the default engine. Runbook: docs/operations/archv2-destination-batching.md.
  • Status writes: failures log pipeline.status_persist_failed and increment
    conduit_pipeline_status_persist_failures_total. It should stay at 0; when it rises, look at the
    database.

Fixes

  • Source teardown waited 10s after a plugin stream ended (#2900, #2947).
  • arch-v2 destination batching stalled (#2979); now refused (#2987).
  • generate.provider was not read (#2908) and --max-retries showed 0 (#2933); fixed in #2923.
  • --help listed wrong exit codes (#2907, #2920).
  • A cancelled call could wedge a standalone WASM processor (#2981, #3039).
  • conduit run --dev restarted its debounce window twice; no apply was dropped (#2995).
  • Also: arch-v2 fan-out record sharing (#2946), pinned release tooling (#2919), image SBOMs (see
    Verify this release), docs (#2982, #2984).

Known issues

  • Secrets passed as ${VAR} in a pipeline config file are stored in plaintext in Conduit's
    database (#2988).     The expanded value is stored in badger, SQLite or Postgres and in every backup; the API redacts it, the store
    does not. Restrict access to the database files and treat the database and its backups as secret.
    A fix is being designed for v0.21; existing databases will need a migration or a re-provision.
  • Postgres v0.14.4:
    • An UPDATE leaving a TOASTed column unchanged halts the source under the default replica
      identity with Avro on. Workaround: REPLICA IDENTITY FULL (conduit-connector-postgres#383).
    • A NULL in a nullable column can fail Avro encoding (conduit-connector-postgres#326).
    • A key-changing UPDATE leaves a duplicate at the destination (conduit-connector-postgres#386).
    • Zero numeric reads 0 in snapshot, 0.000 in CDC (conduit-connector-postgres#391).
    • NULL uuid is emitted as "" (conduit-connector-postgres#392).
    • int8 max as a JSON number is stored as its minimum on amd64 (conduit-connector-postgres#402).
    • A cancelled write can still be applied by the server. It is not acked and is redelivered, so a
      destination table without a key can get a duplicate row.
  • S3 v0.9.5:
    • CDC misses uploads longer than lookbackPeriod and reports no deletes in unversioned buckets.
    • The snapshot's last object is delivered again (conduit-connector-s3#1040).
    • A restart during the snapshot starts it over (conduit-connector-s3#1041).
    • Static keys only, no AWS default credential chain (conduit-connector-s3#1021); see #2988.
    • A real AWS regional aws.endpoint logs a false warning (conduit-connector-s3#1022).
  • arch-v2:
    • The batching refusal at boot is only logged; the pipeline shows stopped with no error (#2989).
    • A stopping pipeline can report the wrong teardown error; nothing is lost (#2966).
    • Destinations buffering outside the SDK can still stall; SIGTERM does not stop them (#2990).
  • Default engine: a source fetcher can spin after its node stops; nothing is acked or dropped (#2976).
  • Status writes are not bounded on badger, the default store (#2964).
  • No chaos test for a source failure racing a graceful stop (#2977).
  • pipeline.shutting_down rarely reaches API clients; they see common.unavailable (#2918).
  • Concurrent config writes on one pipeline can leave orphans. Avoid them (#2992).
  • Advisories GO-2026-5046/5047/5048 (via the Postgres connector's hamba/avro/v2) and
    GO-2026-5932 (golang.org/x/crypto/openpgp) have no fix and are in the govulncheck baseline.

Deferred

  • A single pipeline engine (v0.21, design #2971). Until then, the RAG template needs
    --preview.pipeline-arch-v2.
  • Debezium-compatible Postgres CDC output (design #3033) and Protobuf decode → v0.21.
  • The v0.22 items in the v0.20.0 notes are unchanged.

Verify this release

Requires cosign v3 or later.
Download checksums.txt, checksums.txt.sigstore.json and your files into one directory:

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json \
  checksums.txt

# only once the signature verifies (on macOS: shasum -a 256 --check --ignore-missing)
sha256sum --check --ignore-missing checksums.txt

The Docker image (pin the digest it prints):

cosign verify \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/conduitio/conduit:v0.20.1

The SBOM is attested on each platform digest, not the tag or index:

DIGEST=$(docker buildx imagetools inspect --raw ghcr.io/conduitio/conduit:v0.20.1 \
  | jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "arm64") | .digest')
# or: DIGEST=$(crane digest --platform linux/arm64 ghcr.io/conduitio/conduit:v0.20.1)

cosign verify-attestation \
  --type spdxjson \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  "ghcr.io/conduitio/conduit@$DIGEST" \
  | jq -r '.payload | @base64d | fromjson | .predicate' > conduit-linux-arm64.spdx.json

Use amd64 for the other platform. Against the tag or index, verify-attestation fails with
none of the attestations matched the predicate type; that is expected.

Changelog, artifacts and verification (generated by GoReleaser)

Changelog

Release Artifacts

OSX Darwin

Linux

Windows

Verifying

checksums.txt is signed with Sigstore keyless signing (checksums.txt.sigstore.json), and each archive has an SPDX SBOM. The Docker image is signed by digest. See Verifying a release.

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json checksums.txt
sha256sum --check --ignore-missing checksums.txt

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.20.1

conduit-release-view

Contributors

devarismeroxa
Assets 24
Loading