Driver does not recover after TOPIC_AUTHORIZATION_FAILED

[bpinhosilva]

Describe the bug
Following similar issue (#938), the driver does not recover after getting TOPIC_AUTHORIZATION_FAILED as described as following:

To Reproduce
Have kafka server running with autoCreateTopic set to true but with an ACL denying creation of specific topic format.

1. Try to send a message using invalid topic (not allowed by ACL), the response would be TOPIC_AUTHORIZATION_FAILED.
2. Try to send a message using valid topic (already present on kafka), the response would be error, once it was not able to clean the targetTopics list from previous trial.

Expected behavior
The driver has to be able to recover to its safe state after receiving topic related errors when pushing messages to kafka.

Observed behavior
After the first error, the driver keeps the invalid topic inside targetTopics causing failures to any subsequent calls to push message to kafka.

Environment:

• KafkaJS version latest
• Kafka version 2.4
• NodeJS version LTS

Solutions

1. We can add this error check (another statement inside if clause) and clean the list if we find them;
2. We can recover the list after any error trying to push messages to kafka.

There are some implications to the 2nd solution. Say for example the publisher is trying to push a message to invalid topic and there is network issue during the operation. The invalid topic would be present in targetTopics causing failure to subsequent calls not matter if they include valid or invalid topic.

Questions?

Is there a situation where we do not want to clean targetTopics ?

kafkajs/src/cluster/index.js

Line 195 in d13acd5

this.targetTopics = previousTopics

Would it cause much trouble if we clean this list no matter what error we get while calling await this.refreshMetadata()?

kafkajs/src/cluster/index.js

Line 192 in d13acd5

await this.refreshMetadata()

Should we add every possible error to the if statement so we just recover the list on those cases?

The fix is underway but I would like to get some input about this topic before submitting the request.

[jseparovic]

I'm hitting this issue too. Is there a workaround?

[bpinhosilva]

this

Another deployment to force topic reload.