Skip to content

Kafka emitter wasn't given the correct number of threads. #15719

Merged
jon-wei merged 1 commit intoapache:masterfrom
TSFenwick:feature-kafka-emitter-has-correct-number-of-threads
Jan 18, 2024
Merged

Kafka emitter wasn't given the correct number of threads. #15719
jon-wei merged 1 commit intoapache:masterfrom
TSFenwick:feature-kafka-emitter-has-correct-number-of-threads

Conversation

@TSFenwick
Copy link
Contributor

@TSFenwick TSFenwick commented Jan 18, 2024

Fixes issue with kafka emitter not having the correct number of threads

Description

The kafka emitter didn't have the correct number of threads. It should be 1 thread per scheduled task. If the kafka emitter was emitting every eventType the lost event logging callable would never get called.

This change intelligently provisions the correct number of threads per scheduled task. 1 for each event type, and 1 for logging the lost events. This is a change to make this work. But in the future it would be worthwhile to make each task not be greedy and share threads so there isn't a need of a thread per task.

Release note

Fixed: KafkaEmitter will now run all scheduled callables. It now intelligently provision threads to make sure there are no wasted threads, and all callables can run.


Key changed/added classes in this PR
  • KafkaEmitter
  • KafkaEmitterConfig

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

…e 1 thread per scheduled task.

This change intelligently provisions the correct number of threads per scheduled task. 1 for each event type, and 1 for logging the lost events.
This is a change to make this work. But in the future it would be worthwhile to make each task not be greedy and share threads so there isn't a need of a thread per task.
@jon-wei jon-wei merged commit 18d42ca into apache:master Jan 18, 2024
@317brian 317brian added the Bug label Feb 28, 2024
@adarshsanjeev adarshsanjeev added this to the 30.0.0 milestone May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants