How to handle UnknownProducerIdException of trxnal producer?

[gelihu]

I am using dotnet-client.

Sometimes producer fails with a message like this:

ProduceRequest for topic[0] with 1 message(s) failed due to unknown producer id (broker 1001 PID{Id:4000,Epoch:0}, base seq 5, 0 retries): unable to retry without risking duplication/reordering

Server log shows following:

[2021-04-06 10:02:35,715] ERROR [ReplicaManager broker=1001] Error processing append operation on partition topic-0 (kafka.server.ReplicaManager) org.apache.kafka.common.errors.UnknownProducerIdException: Found no record of producerId=4000 on the broker at offset 900in topic-0. It is possible that the last message with the producerId=4000 has been removed due to hitting the retention limit.
[2021-04-06 10:04:24,485] INFO [TransactionCoordinator id=1001] Completed rollback ongoing transaction of transactionalId: client-4743a13e-501e-4d07-b2ea-7e4b492d0ccc-id due to timeout (kafka.coordinator.transaction.TransactionCoordinator)
[2021-04-06 10:04:47,776] INFO [TransactionCoordinator id=1001] Initialized transactionalId client-1e43124b-f161-490d-85d2-542ee40a08f8-id with producerId 4001 and producer epoch 0 on partition __transaction_state-21 (kafka.coordinator.transaction.TransactionCoordinator)

After searching for a while I found out about (KIP 360)[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820]. It doesn't appear to be supported by this client yet.

What would be good a workaround till then?

[edenhill]

The upcoming 1.7.0 release will have KIP-360 support.
In the meantime there is no safe way for an application to know if the message was persisted or not, so the best option now is to decomission the producer and create a new - it is your decision if you want to send the same message again or not.