-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
advice for the erlang binding #3300
Comments
Perhaps in step 2 you exceeded The contract is for the application to call poll() at least every max poll interval so that it can detect rebalances and stop processing messages for partitions that have been revoked and assigned to other consumers. What you can do is separate the consumer queue (where rebalance events, errors, etc are emitted) and the partition fetch queues (where messages are emitted), and then just make sure to poll the consumer queue while doing processing so that you can detect rebalances. |
Yes you are right ! in step 2 I exceed the Right now my implementation does the following: on
Inside my erlang code each partition is handled by an erlang process that:
Can you please let me know how I can also forward consumer queue to another queue to avoid the problems |
With all partition queues re-routed you should be able to just call consumer_poll() regularily from a separate thread to serve max.poll, rebalances, etc. |
hmm, from what I see my logic with the main queue is as follow:
And when I picked this implementation to optimize the calls to *_poll functions here. But seems is not covering Instead doing this I think the easy fix will be to make sure that calls between |
Yes, that sounds like a good plan, |
@edenhill thanks a lot ! will do the changes and see if problem improves ! Silviu |
Hello,
Into https://github.com/silviucpp/erlkaf we are using a design where on each topic we have a dedicated erlang process that calls rd_kafka_queue_poll on the partition queue on regular intervals to get new events.
Problem we faced recently :
The messages had to be sent into a system which was down for a very long time. Our consumer is doing something like:
As you can see in step 2 you can stay indefinitely amount of time while you are not calling any API on that queue..
This seems causing issues, for some reason the consumer didn't reconnected and the offsets were lost because on the server were set for 24 hours.
My question is: what API we can call to ping the librdkafka that we are still alive ?
Or if we call rd_kafka_queue_poll(..) and get message A B C, then A and B are processed completely and we call rd_kafka_offset_store with B offset, once we calling again
rd_kafka_queue_poll we will get starting from C (I mean C was already returned on the previous call also but we couldn't proces it).
Silviu
The text was updated successfully, but these errors were encountered: