This repository has been archived by the owner on Jan 24, 2024. It is now read-only.
Issue 95: Make Kafka OffsetCommit and Pulsar backlog align #105
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jiazhai
force-pushed
the
jia/offset_commit_cursor
branch
from
33f81d1
to
60f9923
Compare
jiazhai
force-pushed
the
jia/offset_commit_cursor
branch
from
bfc58c3
to
afb1a90
Compare
jiazhai
force-pushed
the
jia/offset_commit_cursor
branch
from
dec3e10
to
0a4bc0f
Compare
jiazhai
force-pushed
the
jia/offset_commit_cursor
branch
from
8fce032
to
2848835
Compare
Merged
BewareMyPower
added a commit
that referenced
this pull request
May 21, 2021
Fixes #392 ### Motivation #105 introduced the `OffsetAcker` to reuse the backlog stat of Pulsar subscription. However, after #296 introduced the continuous offset, when the `OffsetAcker` want to find the `MessageId` or `Position`, which contains the ledger id and entry id of a batch, it needs to get the `PersistentTopic` using `BrokerService#getTopic`. If the current broker was not the owner broker of the partition, `getTopic` would fail. However, Kafka consumers send the `OFFSET_COMMIT` requests to the coordinator broker, which is associated with the specified group name. A coordinator broker may not own the partition after the pulsar's topic ownership changed, so the `getTopic` would fail and couldn't be recovered, which affect the performance significantly and a lot of warnings logs would be printed. ### Modifications - Remove the `OffsetAcker` with the related metrics (`CoordinatorStats`) and tests (`CommitOffsetBacklogTest`) - Remove the reused consumer stats from #263 with the related tests (`testKafkaConsumerMetrics`) - Add `testMixedConsumersWithSameSubscription` to cover the case that Kafka consumer and Pulsar consumer subscribes the same topic with same subscription name. Because before this PR, Pulsar consumer were aware of the Kafka consumer's consumed position (committed offset). - Fix the current tests: - `testDeleteTopics`: After this PR, the topic could be deleted even the Kafka consumers were still connected because no broker side Consumer were attached to the topic. - `testPulsarProduceKafkaConsume2`: It's a test for Kafka consumer with `enable.auto.commit=true`, however the validation is wrong because even a consumer hasn't committed any offset, the consumer can still consume from the latest offset. Regarding to the cases that Kafka or Pulsar producer produces and Kafka or Pulsar consumer consumes, they are covered by existed `KafkaRequestTypeTest`. And the offset commit case is covered by `DifferentNamespaceTestBase#testCommitOffset`.
Demogorgon314
pushed a commit
to Demogorgon314/kop
that referenced
this pull request
Oct 30, 2023
Master issue: streamnative/ksn#63 ### Modifications For `SASL_*` requests, forward them to any broker (`authenticateBroker`) to verify if the authentication succeeded. At the same time, cache these requests in `BrokerConnectionGroup` because we might need to forward further requests (e.g. `PRODUCE`) to a new connection, in this case, we need to forward the `SASL_*` requests again so that the `KafkaRequestHandler#authenticator` will have the principal info for authorization. `SaslPlainProxyTest` is added to verify this patch. In addition, close the connections to broker in `BrokerConnectionGroup` when the client connection to proxy is closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #95.
Make Kafka OffsetCommit and Pulsar backlog align.