Skip to content

Docs: Document Kafka Connect control topic purpose and retention#16576

Open
wombatu-kun wants to merge 1 commit into
apache:mainfrom
wombatu-kun:issue/15844-document-control-topic-retention
Open

Docs: Document Kafka Connect control topic purpose and retention#16576
wombatu-kun wants to merge 1 commit into
apache:mainfrom
wombatu-kun:issue/15844-document-control-topic-retention

Conversation

@wombatu-kun
Copy link
Copy Markdown
Contributor

Summary

Users have reported confusion about the Kafka Connect sink's control topic growing without bound (#15844, #15844). The docs explain how to create the control topic but never describe what it is for, how its events are used, or that it should have a finite retention — so on brokers with a large or unlimited default retention.ms, the topic accumulates coordination events indefinitely. This documents the behavior and the recommended configuration.

What changed

Expanded the "Control topic" section of docs/docs/kafka-connect.md:

  • Added a paragraph on the control topic's purpose and the per-commit event flow (StartCommit / DataWritten / DataComplete / CommitToTable / CommitComplete), noting that DataWritten carries data/delete file metadata rather than rows, and that the durable commit position lives in the table snapshot — so control-topic events are transient.
  • Added --config retention.ms=3600000 to the topic-creation example.
  • Added a "Control topic retention" subsection: why an auto-created topic grows under broker defaults, how to size retention.ms relative to the commit interval/timeout, how to set it on an existing topic, multi-connector sizing, and using cleanup.policy=delete rather than compaction.

Closes #15844

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Control topic always increasing

1 participant