You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm trying to migrate rust-rdkafka from the callback based API to the event one.
If we consume messages via the callback API (i.e rd_kafka_consumer_poll), then RD_KAFKA_OP_ERR errors are not passed back to the application while RD_KAFKA_OP_CONSUMER_ERR are. This is visible here
where RD_KAFKA_OP_CONSUMER_ERR will return RD_KAFKA_OP_RES_PASS while RD_KAFKA_OP_ERR will provide the error to any error_cb registered or just log it and return the default result value which is RD_KAFKA_OP_RES_HANDLED.
Later on rd_kafka_consume0 will get this op_result and just skip the return of a message with the RD_KAFKA_OP_ERR error.
If we consume messages via the Event API (i.e rd_kafka_queue_poll on the consumer queue after enabling the events via rd_kafka_conf_set_events), rd_kafka_op2event will set both RD_KAFKA_OP_ERR and RD_KAFKA_OP_CONSUMER_ERR as RD_KAFKA_EVENT_ERROR.
This results on the consumer now passing back any RD_KAFKA_OP_ERR errors to the application rather than just RD_KAFKA_OP_CONSUMER_ERR errors as with the callback API. For example, RD_KAFKA_RESP_ERR__TRANSPORT errors are now passed back to the app.
Is this difference accepted and is this the expected behaviour or is this a bug? I do see confluence-kafka-go has a special case for PartitionEOF but then passes back all errors to the application (with a flag for fatal ones).
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
librdkafka version (release number or git tag): 2.3.0
Apache Kafka version: 2.x and 3.x
librdkafka client configuration: defaults.
Operating system: <REPLACE with e.g., Centos 5 (x64)>
Provide logs (with debug=.. as necessary) from librdkafka
Provide broker log excerpts
Critical issue
The text was updated successfully, but these errors were encountered:
@emasab I know guys this is not a high priority for you, but is there any way where I can easily get the list of transient errors (assuming this is the diff between RD_KAFKA_OP_ERR and RD_KAFKA_OP_CONSUMER_ERR) to filter those in rust-rdkafka when using the event API? we're getting some user reports and we want to filter those errors (i.e stop bubbling transient errors to the users) to rule out some issues. Any pointers will be highly appreciated.
By looking at the code, it looks like we build RD_KAFKA_OP_CONSUMER_ERR errors (i.e errors that are returned to the app) by calling rd_kafka_consumer_err. Then by looking at where this is called, I’ve filtered the following errors as the one needed to be bubble to the users via poll. This is not 100% accurate as I might have missed when there are nested calls, so this is fragile. I'm looking a list like this. The issue described on the ticket only affects the consumer. It does not affect the producer.
Description
I'm trying to migrate rust-rdkafka from the callback based API to the event one.
If we consume messages via the callback API (i.e
rd_kafka_consumer_poll
), then RD_KAFKA_OP_ERR errors are not passed back to the application while RD_KAFKA_OP_CONSUMER_ERR are. This is visible herelibrdkafka/src/rdkafka.c
Lines 3846 to 3869 in 95a542c
Later on
rd_kafka_consume0
will get this op_result and just skip the return of a message with the RD_KAFKA_OP_ERR error.librdkafka/src/rdkafka.c
Lines 3134 to 3150 in 95a542c
If we consume messages via the Event API (i.e
rd_kafka_queue_poll
on the consumer queue after enabling the events viard_kafka_conf_set_events
),rd_kafka_op2event
will set both RD_KAFKA_OP_ERR and RD_KAFKA_OP_CONSUMER_ERR as RD_KAFKA_EVENT_ERROR.librdkafka/src/rdkafka_event.h
Lines 40 to 41 in 95a542c
This results on the consumer now passing back any RD_KAFKA_OP_ERR errors to the application rather than just RD_KAFKA_OP_CONSUMER_ERR errors as with the callback API. For example, RD_KAFKA_RESP_ERR__TRANSPORT errors are now passed back to the app.
Is this difference accepted and is this the expected behaviour or is this a bug? I do see confluence-kafka-go has a special case for PartitionEOF but then passes back all errors to the application (with a flag for fatal ones).
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
<REPLACE with e.g., Centos 5 (x64)>
debug=..
as necessary) from librdkafkaThe text was updated successfully, but these errors were encountered: