Consumers unable to recover upon kafka cluster failure

[vineetgoel]

I noticed that my application which uses both producers and consumers was unable to recover upon some boxes failing in my kafka cluster. The errors I see in the logs are as follows (with errors closest to the point until which my application was running fine, on top):

• kafka.errors.GroupCoordinatorNotAvailableError: [Error 15] GroupCoordinatorNotAvailableError
• [2017-12-04 10:28:30,621: INFO] OffsetCommit failed for group foo due to a coordinator error (RequestTimedOutError), will find new coordinator and retry
• kafka.errors.RequestTimedOutError: [Error 7] RequestTimedOutError

Not sure why the consumers were unable to recover at this point. What is the ideal way of handling such exceptions within the consumer. It also seems like the consumers don't really raise this exception and only logs them.

[vineetgoel]

Verified that the consumer is able to recover on in most cases.

However there are times when the following error is raised and the consumer crashes:

kafka.errors.GroupCoordinatorNotAvailableError: [Error 15] GroupCoordinatorNotAvailableError

[tvoinarovskyi]

Should be fixed by #286.