Embrace auto.topic.create; trust defaults for production topics #107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
so that we can omit replication-factor when creating topics, as we benefit from reusing such commands or definitions across dev-qa-prod.
topics at runtime, such as splitting a stream based on some business enum. Producers and Kafka Streams apps would otherwise need to set up an AdminClient to do that. This reverts commit: 0681cc5
and easy to reconfigure per topic as they grow. We already had that, but this branch cares about grouping such conf. It also encourages topic defaults geared towards persistent data.
config location suggested by Kafka's sample conf.
as we now encourage close scrutiny of the config file.
log.dir = /tmp/kafka-logs log.dirs = /var/lib/kafka/data/topics but this is the lesser of two evils compared to duplicate values
solsson
added a commit
that referenced
this pull request
Dec 18, 2017
This was referenced Dec 31, 2017
6 tasks
solsson
added a commit
that referenced
this pull request
Nov 28, 2018
so we want to undo #107. It was partially based on a false assumption, as pointed out in #101 (comment) Topics are created not only at produce but also at for example kafkacat -C. Typos cost us more time than it would take to automate topic creation and run ./bin/kafka-topics.sh in a temporary pod when we haven't automated.
solsson
added a commit
that referenced
this pull request
Dec 20, 2018
solsson
added a commit
that referenced
this pull request
Oct 8, 2019
I though that the feature would be neat for development (with replication factor 1, see #222) but it causes just as much confusion and useless troubleshooting there, for example race conditions between intentional topic creation and a container starting up to produce to the topic. You actually never know which topic config you're getting. Related: #107 The duplication is a workaround for kubernetes-sigs/kustomize#642
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a mindset change for us, meant to address at least one bullet in #101: we will omit the CLI args for number of replicas and partitions when creating production topics. Instead the defaults should reflect the current cluster size - and stage. Test clusters may replicate less, in order to save storage cost for example.
Producers still need to configure
acks, but broker config helps here too. We should setallby default (or-1if it has to be an int), and let min.insync.replicas be configured so that single node loss is a non-event.Our defaults will be for stability. If some topics need better througput (#88, #92 ?) we'll explicitly create (or reconfigure) those.