Make it possible to roll Kafka or ZooKeeper pods individually

[scholzj]

Type of change

• Enhancement / new feature

Description

Strimzi currently supports use of annotation on the StatefulSet resources to trigger manually a rolling update. But in some cases, it might make sense to roll just some of the pods but do it through the operator instead of deleting them manually. Doing it through the operator can ensure that:

• The manual deletion does not conflict with any action the operator might take at the same time (e.g. delete another pod in parallel)
• Use the operator logic to do the roll (especially important for Kafka where it checks the in-sync replicas etc.)

This PR makes it possible by using the same annotation as for StatefulSet, but on the pod level. When multiple pods from the same StatefulSet are annotated, they will be rolled one after another but within the same reconciliation run.

System tests and documentation will be done in a separate PR.

Checklist

• Write tests
• Make sure all tests pass
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Update CHANGELOG.md

[scholzj]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).