processor gets stuck in shutdown after error

[frairon]

We have seen cases where a running processor gets stuck while shutting down, when one of the brokers goes down or the metadata is changed due to e.g. rebalance.
The error is supposedly a kafka.Promise that is never finished, because the producer gets cancelled. This leads to the waitgroup in https://github.com/lovoo/goka/blob/master/partition.go#L142 to never get done, so the partition never shuts down, thus the whole processor gets stuck.
We could not reproduce the bug, but a stack trace of one of the stuck processors showed this location.

PR is open, hope that will fix the issue.

[db7]

The producer does not get canceled on rebalance. I think it should be then related to some error on emitting a message. Do you have the stack trace?

[frairon]

Sorry, my bad. The example was just wrong.
What happens is in this order:

• a broker goes down
• the leader of some partitions is not availble anymore
• emitting fails (and sometimes the error is, that metadata has changed)
• that makes the processor close the producer
• the producer stops processing the Success/Error channels
• the message processing is incomplete, the waitgroup is never finished
• the partition cannot shutdown
• the processor is stuck