Skip to content

v0.20.2

Choose a tag to compare

View all tags
@github-actions github-actions released this 15 Oct 11:32
· 165 commits to main since this release
v0.20.2
74d1b8f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Conduit v0.20.2

v0.20.2 is a patch release. It fixes a bug where avro.encode could silently write wrong data when
two pipelines in one Conduit process use schemas that share type names. It also fixes four security
issues, and adds a guard so that v0.20.2 refuses to run pipelines stored by v0.21. Embedded libraries:
conduit-commons v0.7.0 → v0.7.1, conduit-schema-registry v0.2.6 → v0.2.7. The built-in connectors are
unchanged (postgres v0.14.4, s3 v0.9.5, kafka v0.12.6).

Upgrading from v0.20.0? Read the v0.20.1 notes
first. Their upgrade and downgrade warnings still apply.

Upgrade if you encode Avro

avro.encode could write records with another schema's encoder (conduit-commons#305,
conduit-commons#308, GHSA-hx9g-59x7-9h4x). Affected: v0.7.0 through v0.20.1: avro.encode since v0.9.0,
and the encodewithschemakey / encodewithschemapayload processors in v0.7 and v0.8, which use the
same code. There is no error or warning, and the records are acked.

It happens when one Conduit process encodes with two schemas that both define a named type (a
record, enum or fixed, including a decimal fixed) with different definitions, and refer to it
by name inside an array, map or union. Typical cases are two pipelines whose subjects share type
names, or two versions of one subject. The second schema is encoded with the first schema's encoder:

  • record fields are written in the other schema's order;
  • enum values point at the wrong symbol;
  • decimals are written at the wrong scale (1.5 becomes 15000).

Which pipeline gets the wrong encoder depends on which one encodes first, so a restart can move the
corruption rather than clear it. Built-in connectors that supply their own schemas go through the
same encoder and are exposed the same way. Decoding is not affected. A single self-contained schema,
and the schemas the connector SDK extracts on its own, never trigger it.

What to do:

  • Upgrade to v0.20.2. Each schema now encodes with its own encoder cache, and the bytes for
    schemas that never collided are identical.
  • Records written before the upgrade are not repaired. If your schemas match the conditions above,
    decode a sample of the output with each record's own schema and look for swapped fields, wrong
    enum symbols, or decimals off by a power of ten.
  • Standalone connectors and processors carry their own copy of conduit-commons. They are exposed
    only if they encode colliding schemas in their own process. Rebuild them against conduit-commons
    v0.7.1 or v0.8.1.
  • If you cannot upgrade yet: give every named type a full name that is unique to its schema (for
    example a namespace per subject and version), or run pipelines with colliding schemas in separate
    Conduit processes.

A Conduit regression test with two pipelines using avro.encode with shared type names is in
#3106.

Security fixes

Details are in each advisory.

  • Schema identity (GHSA-94xc-rjcp-83cr, GHSA-pfr7-p89q-w2jp, GHSA-95wx-33xh-gpvw). conduit-commons, the built-in
    schema registry (conduit-schema-registry) and Conduit's schema registry client could treat two
    different schemas as the same schema. Someone who can register schemas in the same registry, for
    example by creating a pipeline, could cause their schema to be used in place of another one. Fixed
    in conduit-commons v0.7.1, conduit-schema-registry v0.2.7 and Conduit (#3106). All
    three fixes are needed together; v0.20.2 ships all three.
  • Standalone plugins inherit Conduit's environment (GHSA-gwvc-9h8p-x5r7). A standalone connector plugin
    receives every environment variable of the Conduit process, including credentials meant only for
    Conduit. v0.20.2 does not change what plugins receive. It names the credential-like variables at
    startup so you can act on them (see below). A future release restricts the plugin environment to
    an allowlist.

Before you upgrade

  • New startup warning for standalone plugins (#3045, #3054, #3065; backport #3084). When
    connectors.path holds at least one standalone connector, Conduit logs one warning per process
    with the code plugin.standalone_env_passed. It lists the names, never the values, of host
    variables that look like credentials and that the plugins will receive. conduit doctor reports
    the same condition as plugins.standalone_env, including in --json; the MCP doctor tool
    reports only a count. Start Conduit with a minimal environment. Once you have reviewed the list,
    silence the warning with connectors.standalone.env-warning: false,
    --connectors.standalone.env-warning=false or CONDUIT_CONNECTORS_STANDALONE_ENV_WARNING=false.
    That only silences the warning. Runbook: docs/operations/standalone-plugin-environment.md;
    site: Standalone plugin environment.
    The allowlist becomes the default in the minor release after this one, and its notes will say so.
  • Downgrade guard for v0.21 databases (#3062). v0.21 stores ${VAR} references in a new
    SettingRefs field and takes them out of the settings. v0.20.2 refuses to start a pipeline whose
    stored connector, processor or DLQ settings carry SettingRefs. The pipeline is marked
    Degraded with pipeline.settings_references_unsupported. Nothing is read or acked, and the
    stored records are left as they are. Without the guard, such a pipeline would run with those settings missing, and an
    optional credential could silently fall back (S3 to the default AWS credential chain, Kafka to no
    SASL). Deleting such a pipeline or connector is refused too, so a connector's cleanup never runs
    without its settings. Heal it by running v0.21 or later, or by re-provisioning it from its
    pipeline file. Runbook: docs/operations/settings-references-unsupported.md. v0.20.0 and
    v0.20.1 do not have this check: do not downgrade from v0.21 to them     unless the pipelines
    directory is mounted and the reconcile succeeds.
  • Avro: a schema that uses a type it does not define now fails to parse (conduit-commons#308).
    Before, it borrowed the definition from whichever schema in the process had defined that name
    first. The same schema already failed in a fresh process. It now fails with
    could not parse avro schema every time.
  • Avro encoders are built once per schema, not once per process. Schemas are cached, so each one
    compiles once on the normal path. This was not benchmarked, and we make no performance claim.
  • Downgrading to v0.20.1 or earlier brings #305 back. Nothing stored changes between v0.20.1 and
    v0.20.2, so downgrading is otherwise as safe as the v0.20.1 notes describe.

Behavior changes

Change v0.20.1 v0.20.2 Code PR
Start a pipeline whose stored settings carry SettingRefs started without those settings refused, Degraded pipeline.settings_references_unsupported, FailedPrecondition #3062
Delete such a pipeline or connector cleanup ran without the settings refused, retried on next boot pipeline.settings_references_unsupported #3062
Standalone plugins with credential-like host variables nothing one startup WARN and a doctor finding plugin.standalone_env_passed #3084
Avro schema using a type it does not define resolved from another schema parse fails — #3106
Avro encode with a colliding named type another schema's encoder own encoder — #3106

New flag: connectors.standalone.env-warning (default true).

Known issues

  • Everything under Known issues in the v0.20.1 notes still applies.
  • A self-referencing Avro schema hangs the pipeline (conduit-commons#314). A schema that refers
    to itself, directly or through another type (a linked list L{next: ["null","L"]}, a tree
    T{kids: array<"T">}), makes schema parsing recurse without bound: the pipeline hangs and
    memory grows until the process dies. One source emitting such a schema is enough. Affects
    v0.20.0, v0.20.1 and v0.20.2. Workaround: do not use self-referencing Avro schemas. Fix in v0.20.3.
  • Some nested Avro unions can silently lose map keys on decode (conduit-commons#312). This
    affects avro.decode and the connector SDK's schema middleware, so destinations that decode Avro
    are affected too. It happens when a union has a record, enum, fixed, map or array
    branch that itself contains another union, for example ["null", map<["null","int"]>]. Map keys
    can be dropped with no error, and some values can stay in their union wrapper; one shape fails
    with expected single value encoded as a map instead. Workaround: avoid such schemas. Fix in
    v0.20.3, which follows shortly.
  • Open conduit-commons Avro issues: a decoded nullable record or map does not re-encode with its own
    schema (conduit-commons#306); local-timestamp-* values go through time.Local
    (conduit-commons#307).

Verify this release

Requires cosign v3 or later.
Download checksums.txt, checksums.txt.sigstore.json and your files into one directory:

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.2' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json \
  checksums.txt

# only once the signature verifies (on macOS: shasum -a 256 --check --ignore-missing)
sha256sum --check --ignore-missing checksums.txt

The Docker image (pin the digest it prints):

cosign verify \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.2' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/conduitio/conduit:v0.20.2

The SBOM is attested on each platform digest, not the tag or index:

DIGEST=$(docker buildx imagetools inspect --raw ghcr.io/conduitio/conduit:v0.20.2 \
  | jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "arm64") | .digest')

cosign verify-attestation \
  --type spdxjson \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.2' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  "ghcr.io/conduitio/conduit@$DIGEST" \
  | jq -r '.payload | @base64d | fromjson | .predicate' > conduit-linux-arm64.spdx.json

Use amd64 for the other platform.

Changelog, artifacts and verification (generated by GoReleaser)

Changelog

Release Artifacts

OSX Darwin

Linux

Windows

Verifying

checksums.txt is signed with Sigstore keyless signing (checksums.txt.sigstore.json), and each archive has an SPDX SBOM. The Docker image is signed by digest. See Verifying a release.

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.2' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json checksums.txt
sha256sum --check --ignore-missing checksums.txt

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.20.2

conduit-release-view

Contributors

devarismeroxa
Assets 24
Loading