Fail fast on librdkafka connection error

[mcandre]

When a Kafka connection fails in librdkafka, this information should propagate up to confluent-kafka-go, so that applications can decide to fail fast, reconnect, etc.

Right now, librdkafka reports connection errors to the console, but the confluent-kafka-go application continues as if everything is just fine.

[edenhill]

Connection errors are treated as temporary errors and librdkafka will retry connecting to its known brokers indefinately.
Given that the application is configured with a correct broker list it is advised for an application not to take any action when a broker goes down since it will most likely come back up sooner or later, or another broker will assume leadership of the given partitions, so the library and application will eventually pick up where it left off.

Instead you should try to define your application-specific constraints in time:

• how long is it okay for a consumer application not to see any new messages?
• how long may the producer try to deliver a message for the application? (message.timeout.ms)

And let the underlying client operate without interruption within those bounds, abstracting your application from Kafka specific know-how.

Anyway, if you're not buying into this, or maybe just curious, you can look for the kafka.Error event.
You will be interested in the ErrTransport and ErrAllBrokersDown events.

[mcandre]

Thanks for the detailed explanation, that's a good way to treat these errors.

[shreyas-srinivas]

@mcandre What if the broker is not down but the client is unable to connect due to a misconfiguration of the broker address. Unless we get a failure event it's not possible for us to know about the problem.

[xin-au]

librdkafka will retry connecting to its known brokers indefinately.
Hi @edenhill , wondering is there a way to config it? Like how often to retry connecting to broker?