Error: Topic XXXX not present in metadata after 60000 ms

[fedeostrit]

hello everyone, am having a problem that occurs randomly at the time when a producer tries to generate messages to any topic, this is the error that the producer receives and I do not see errors in Kafka's logs:

Error: Topic XXXX not present in metadata after 60000 ms

This can be given in any topic and at any time, there is no time pattern and the connection to kafka is stable, also the configuration in kafka to automatically create a topic is enabled auto.create.topics.enable: "true"

I leave the configuration of the producer that is being used to create messages:

retries=2700000
retry.backoff.ms=1000
batch.size=131072
buffer.memory=66554432
bootstrap.servers=Ipserver:9094
compression.type=snappy
acks=all
linger.ms=100
request.timeout.ms=60000
enable.idempotence=true
delivery.timeout.ms=180300
metadata.max.idle.ms=180000

I
cluster.zip
have a cluster of 3 nodes in azure AKS and attach the cluster.yaml in case you need to see the config of it

[scholzj]

I do not think this error is sent from broker. This error says that the topic which the producer is trying to use is not present in the metadata it requested / received from the cluster. I'm not sure it is easy to answer why do you get it. Could be some connectivity issue? Does the topic it complains about actually exist? Or is it just being created? Etc.

[fedeostrit]

Hi Jakub good morning,

When the producer fails, the Topics are already created because of what already exists, I do not think it is a connectivity problem such as a drop in the VPN because there are no signs of drop on the monitors

When there is a break in communication and the VPN is cut in the producer, another error is received from what you can see and it is the following:

An error occurred during the conversation with Kafka.
Error: Expiring XXXX record(s) for XXXXX-0:180300 ms has passed since batch creation

The problem is very strange because the producers are constantly sending messages to the Kafka cluster, from different databases they are sending the changes that are made in the tables using CDC, the failures occur randomly with the commented error and the producer ends his execution when rebooted after error keeps sending messages without problems

can it be some setting in the Producer parameters that can be added or changed to test if the behavior changes?

in a blog I saw that they added these parameters but I don't know if it would be good to apply them

{ "key": "connections.max.idle.ms", "value": "60000" }, { "key": "metadata.max.idle.ms", "value": "60000" }, { "key": "metadata.max.age.ms", "value": "60000" }

[ahu0605]

I also have the exception
1、I search controller.log,server.log,gc.log, can not find leader election and rebalance
2、I find my app log ,I think maybe batch bytes too large so the i/o thread block
but I have no idea

[fedeostrit]

Hi ahu0605, changing the configuration of the kafkaproducer.properties in the producers and adding the configurations that were described above I was able to achieve that this error does not occur:

{ "key": "metadata.max.idle.ms", "value": "60000" }, { "key": "metadata.max.age.ms", "value": "60000" }

this is my full kafkaproducer.properties:

retries=2700000
retry.backoff.ms=1000
batch.size=131072
buffer.memory=66554432
bootstrap.servers=XX.XX.XX.XX:9094
compression.type=snappy
acks=all
linger.ms=100
request.timeout.ms=60000
enable.idempotence=true
delivery.timeout.ms=180300
metadata.max.idle.ms=60000
metadata.max.age.ms=60000
max.block.ms=180000

[ahu0605]

so do I, will observe the app one week!

[SauravCP]

I know this is pretty late, but seems the spring property spring.kafka.bootstrap-servers changed to spring.kafka.bootstrap.servers attributing to the issue. I could resolve my errors with just this change alone.

[0019]

This helped me. I got misspelling in the name spring.kafka.bootstrap-servers.

[yashtailor]

Hi @fedeostrit I know its pretty late but how did you verify that its working fine? Because this issue is an intermittent one. Did you try reproducing the issue and it didn't come after those changes? Ideally, max.block.ms default value is 60 seconds, the call would anyways fail if the broker was not available for 60 seconds.