Send Kafka a TERM signal at pod stop and wait for shutdown

[solsson]

Fixes #206.

Based on https://github.com/apache/kafka/blob/trunk/bin/kafka-server-stop.sh and https://github.com/apache/kafka/blob/trunk/bin/zookeeper-server-stop.sh but these scripts don't wait for shutdown to complete.

Got the wait loop from https://stackoverflow.com/questions/17894720/kill-a-process-and-wait-for-the-process-to-exit

Currently I've only tested the PR locally with little load. @stigok Can you confirm that you no longer get corrupted indices?

[solsson]

The last log entry I see is INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)

[solsson]

Maybe Zookeeper doesn't need controlled shutdown. I see no effect in logs of invoking the script.

[stigok]

I'm unable to reproduce the bad indices. I don't know how I ended up with them in the first place. We've been having a lot of pod restarts and failed probes running in AKS, so it could've been caused by a lot of different factors.

[stigok]

But this is PR is certainly a step in the right direction 👍

[stigok]

I had bad indexes again after my disks went full. Maybe that is a "good way" to simulate broken indexes.

• Configure log.retention.bytes to a value greater than available disk-space
• Produce enough messages to fill the disk
• Watch Kafka die
• Expand disk and expect to see bad indexes