Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix kafkasql node creation on slow machines with already existing messages. #2991

Closed
carlesarnal opened this issue Nov 30, 2022 · 2 comments
Closed
Assignees
Labels
area/storage Bug Something isn't working

Comments

@carlesarnal
Copy link
Member

For slower compute nodes, the internal H2 DB that we use might not be ready at the time that the Kafka consumer thread starts. This results in consuming existing messages from the Kafka topic before the internal DB is ready to store them.

The workaround is to increase the startup delay of the Kafka message consumer thread.

We will be attempting to fix this problem with some sort of thread notification strategy (where the thread that creates the H2 internal DB will notify the Kafka consumer thread when it's ready to go).

@carlesarnal carlesarnal added the Bug Something isn't working label Nov 30, 2022
@carlesarnal carlesarnal self-assigned this Nov 30, 2022
@EricWittmann
Copy link
Member

The decision on this was to make the initialize logic in the abstract SQL storage available as a CompletableFuture. This will allow the kafka storage impl to chain together the DB initialization with the startup of the Kafka consumer thread.

@EricWittmann
Copy link
Member

Change of plan: I decided to use CDI events to solve this problem:

#3058

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants