Skip to content

cdc: support top-level compression option for kafka#166390

Open
techcodie wants to merge 4 commits intocockroachdb:masterfrom
techcodie:fix-kafka-compression
Open

cdc: support top-level compression option for kafka#166390
techcodie wants to merge 4 commits intocockroachdb:masterfrom
techcodie:fix-kafka-compression

Conversation

@techcodie
Copy link
Copy Markdown
Contributor

Description

The kafka sink historically supported the "Compression" option inside the kafka_sink_config JSON object, but rejected compression as a top-level option. This created inconsistency and confusion, as documented in #113495.

This commit allows using the compression top-level option with the kafka sink. It reads the top-level option (if present) and applies the correct codec to the sarama configuration, making it align with how other sinks receive compression settings.

Fixes #113495

Release Notes

  • changefeedccl: improved the kafka sink to accept the top-level compression option in CREATE CHANGEFEED statements, alongside the kafka_sink_config option.

The kafka sink historically supported the "Compression" option inside
kafka_sink_config, but rejected "compression" as a top-level option.
This commit allows using the "compression" top-level option with
the kafka sink. It reads the top-level option and applies the correct
Codec to the sarama config.

Fixes cockroachdb#113495
@techcodie techcodie requested a review from a team as a code owner March 23, 2026 04:02
@techcodie techcodie requested review from jeffswenson and removed request for a team March 23, 2026 04:02
@trunk-io
Copy link
Copy Markdown
Contributor

trunk-io Bot commented Mar 23, 2026

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

@blathers-crl
Copy link
Copy Markdown

blathers-crl Bot commented Mar 23, 2026

Thank you for contributing to CockroachDB. Please ensure you have followed the guidelines for creating a PR.

Before a member of our team reviews your PR, I have some potential action items for you:

  • Please ensure your git commit message contains a release note.
  • When CI has completed, please ensure no errors have appeared.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl Bot added the O-community Originated from the community label Mar 23, 2026
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@blathers-crl
Copy link
Copy Markdown

blathers-crl Bot commented Mar 23, 2026

Thank you for updating your pull request.

Before a member of our team reviews your PR, I have some potential action items for you:

  • We notice you have more than one commit in your PR. We try break logical changes into separate commits, but commits such as "fix typo" or "address review commits" should be squashed into one commit and pushed with --force
  • Please ensure your git commit message contains a release note.
  • When CI has completed, please ensure no errors have appeared.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@techcodie techcodie marked this pull request as draft March 23, 2026 04:34
@techcodie techcodie marked this pull request as ready for review March 23, 2026 04:34
@blathers-crl
Copy link
Copy Markdown

blathers-crl Bot commented Mar 23, 2026

Thank you for updating your pull request.

Before a member of our team reviews your PR, I have some potential action items for you:

  • We notice you have more than one commit in your PR. We try break logical changes into separate commits, but commits such as "fix typo" or "address review commits" should be squashed into one commit and pushed with --force
  • Please ensure your git commit message contains a release note.
  • When CI has completed, please ensure no errors have appeared.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl
Copy link
Copy Markdown

blathers-crl Bot commented Mar 23, 2026

Thank you for updating your pull request.

Before a member of our team reviews your PR, I have some potential action items for you:

  • We notice you have more than one commit in your PR. We try break logical changes into separate commits, but commits such as "fix typo" or "address review commits" should be squashed into one commit and pushed with --force
  • Please ensure your git commit message contains a release note.
  • When CI has completed, please ensure no errors have appeared.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@techcodie
Copy link
Copy Markdown
Contributor Author

Thanks for the review! I've updated both the v1 (sarama) and v2 (kgo) paths to include a comment clarifying that if both the top-level compression option and the Producer.Compression in kafka_sink_config are set, the top-level option takes precedence.

This keeps the behavior consistent with how other sinks prioritize top-level options over sink-specific JSON configurations. I also fixed the buildKafkaConfig signature in the v1 path which was previously referencing sinkOpts without it being in the parameter list. The updated commit has been pushed to the branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

O-community Originated from the community T-cdc

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cdc: Confusing kafka compression option

3 participants