Migrate Notification Publisher to Confluent Parallel Consumer #586
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It doesn't play well with Kafka Streams. Signed-off-by: nscuro <nscuro@protonmail.com>
Because it does not need any of the features provided by Kafka Streams, it doesn't necessarily make sense to use KS. I looked into migrating to SmallRye messaging (the "native" Quarkus solution), but it still has unpleasant drawbacks that make it not a good fit (#215 (comment)). Most notably, ordered, blocking processing will process all events on a single thread, no matter how many partitions the input topics have. This is prone to become a huge bottleneck under high load. With Parallel Consumer, we get ordering guarantees by key and *still* almost unlimited concurrency (https://github.com/confluentinc/parallel-consumer#ordered-by-key). Further, processing is decoupled from the actual number of partitions. A nice side-effect is also that it supports retries (https://github.com/confluentinc/parallel-consumer#retries). Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro
changed the title
commit 1d6c7f1 Author: Niklas <nscuro@protonmail.com> Date: Mon May 29 11:42:01 2023 +0200 Revise labels in Helm chart (#583) * Properly truncate `name` and `fullname` such that they never exceed 63 characters, even with suffixes * Add `app.kubernetes.io/part-of` label * Make `app.kubernetes.io/name` specific to individual services, instead of using a common label values across all services * Use fully qualified names in `metadata.name` Signed-off-by: nscuro <nscuro@protonmail.com> commit 4f3f6fe Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon May 29 10:40:29 2023 +0100 Bump checkstyle from 10.11.0 to 10.12.0 (#587) commit d209dd2 Author: Niklas <nscuro@protonmail.com> Date: Mon May 29 11:39:32 2023 +0200 Replace Flyway with testcontainers init script (#585)
It doesn't play well with Kafka Streams. Signed-off-by: nscuro <nscuro@protonmail.com>
Because it does not need any of the features provided by Kafka Streams, it doesn't necessarily make sense to use KS. I looked into migrating to SmallRye messaging (the "native" Quarkus solution), but it still has unpleasant drawbacks that make it not a good fit (#215 (comment)). Most notably, ordered, blocking processing will process all events on a single thread, no matter how many partitions the input topics have. This is prone to become a huge bottleneck under high load. With Parallel Consumer, we get ordering guarantees by key and *still* almost unlimited concurrency (https://github.com/confluentinc/parallel-consumer#ordered-by-key). Further, processing is decoupled from the actual number of partitions. A nice side-effect is also that it supports retries (https://github.com/confluentinc/parallel-consumer#retries). Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro
force-pushed
the
migrate-np-to-parallel-consumer
branch
from
8667931
to
a2a9a6f
Compare
…DependencyTrack/hyades into migrate-np-to-parallel-consumer
|
SonarCloud Quality Gate failed. |
|
Passing the Sonar Quality code coverage because coverage seems good locally and is optimal. |
sahibamittal
approved these changes
Jun 7, 2023
nscuro
added a commit
that referenced
this pull request
Jul 2, 2023
As of #586, concurrency of notification publishing is not directly coupled to partition count anymore. For local testing and demo purposes, there's no need to use more than one partition. Signed-off-by: nscuro <nscuro@protonmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because it does not need any of the features provided by Kafka Streams, it doesn't necessarily make sense to use KS.
I looked into migrating to SmallRye messaging (the "native" Quarkus solution), but it still has unpleasant drawbacks that make it not a good fit (#215 (comment)). Most notably, ordered, blocking processing will process all events on a single thread, no matter how many partitions the input topics have. This is prone to become a huge bottleneck under high load.
With Parallel Consumer, we get ordering guarantees by key and still almost unlimited concurrency (https://github.com/confluentinc/parallel-consumer#ordered-by-key). Further, processing is decoupled from the actual number of partitions.
A nice side-effect is also that it supports retries (https://github.com/confluentinc/parallel-consumer#retries).
Relates to #346