feat: expose rackID option in ingester samara client #3395
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Support for KIP-392 in Sarama consumer client This will allow ingester consumers to fetch from the closest replica. On branch add-kafka-consumer-client-rack Changes to be committed: modified: cmd/ingester/app/flags.go modified: cmd/ingester/app/flags_test.go modified: pkg/kafka/consumer/config.go Signed-off-by: Shai Moria <shai.moria@zooz.com> Signed-off-by: shaimoria <shai.moria@zooz.com>
shyimo
force-pushed
the
add-kafka-consumer-client-rack
branch
from
c4ad44d
to
3bc0a3e
Compare
Codecov Report
@@ Coverage Diff @@
## master #3395 +/- ##
==========================================
- Coverage 96.48% 96.47% -0.02%
==========================================
Files 260 260
Lines 15215 15220 +5
==========================================
+ Hits 14680 14683 +3
- Misses 452 453 +1
- Partials 83 84 +1
Continue to review full report at Codecov.
|
yurishkuro
reviewed
Nov 16, 2021
cmd/ingester/app/flags.go
Outdated
| flagSet.String( | ||
| KafkaConsumerConfigPrefix+SuffixRackID, | ||
| "", | ||
| "Configure the 'rack' in which the consumer resides to enable - must be supported by kafka server") |
There was a problem hiding this comment.
Hi @yurishkuro.
i update the description
|
@yurishkuro add explanation for the use case |
set better description for 'rack-id' cli flag Changes to be committed: modified: cmd/ingester/app/flags.go Signed-off-by: shaimoria <shai.moria@zooz.com>
shyimo
force-pushed
the
add-kafka-consumer-client-rack
branch
from
2fc220d
to
824ca28
Compare
yurishkuro
approved these changes
Nov 18, 2021
Merge branch 'master' into add-kafka-consumer-client-rack
…hyimo/jaeger into add-kafka-consumer-client-rack
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: shaimoria shai.moria@zooz.com
Which problem is this PR solving?
It is common to have a Kafka cluster spanning multiple datacenters. For example, a common deployment is within an AWS region in which each availability zone is treated as a datacenter. In order to reduce cross region/rack network cost, we can tell the consumer which rack he's located on. this way the consumer will fetch only from partitions that are located on he's region/rack
Short description of the changes
kafka.consumer.rack-idcli argument in order to ingester consumers to fetch from closest replica