Nakadi producers does not recover from kafka broker shutdowns

[rcillo]

Besides periodically refreshing the list of available brokers https://github.com/zalando/nakadi/blob/nakadi-jvm/src/main/java/de/zalando/aruha/nakadi/repository/kafka/KafkaLocationManager.java#L90-L98 Nakadi does not refresh it's producer instances with up-to-date properties (list of brokers).

Since producers are created during initialisation phase only https://github.com/zalando/nakadi/blob/nakadi-jvm/src/main/java/de/zalando/aruha/nakadi/repository/kafka/KafkaFactory.java#L16, they will not use recently created brokers. It may cause inconsistent behaviour when publishing events.

Nakadi should recover from such scenarios by discovering recently created brokers and updating producers with such information.

[rcillo]

This was a false positive. New brokers weren't getting any traffic due to a bug in the rebalancing of buku that was fixed here https://github.com/zalando/saiki-buku/commit/c37624cbd383e8e29257dc8c7716eaee26c10a6a