Evaluate volume expansion

[solsson]

https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims

[clintfred]

@solsson Have you been able to test this on GKE? I see the volumeClaimTemplate starts with 10Gi instead of 200. Can you characterize the impact of the resize on the Kafka cluster?

Slightly off topic, but perhaps related - in my development cluster, we sometimes end up wanting to completely start with a new cluster (zookeeper, brokers, registry), but retain the data. I have been running delete -f on the directories in reverse order, but after reading #167 and #154 maybe I should have just used apply -f and then deleted the old pods to pick up the new configmaps? It also looks like there is more dynamic config coming in 4.x?

[solsson]

@clintfred

Have you been able to test this on GKE? I see the volumeClaimTemplate starts with 10Gi instead of 200. Can you characterize the impact of the resize on the Kafka cluster?

I only created a new test cluster, but never got round to trying the expansion. If it works we'd create new clusters at 10Gi because that's the minimum PD size (SSDs can be smaller) in GKE. We can never really estimate data volumes at cluster creation. I'll post here if/when we've actually evaluated anything :)

I'll post in #154 regarding cluster replacement.

[clintfred]

In Kubernetes 1.11 release announcement

Support for online resizing of Persistent Volumes has been introduced as an alpha feature. This enables users to increase the size of PVs without having to terminate pods and unmount volume first. The user will update the PVC to request a new size and kubelet will resize the file system for the PVC.

[solsson]

And it looks like offline resize could actually make it to non-alpha GKE clusters in 1.11. Release notes say "Resizing PersistentVolumes after pod restart is now considered beta." with a reference to kubernetes/enhancements#284.

Kafka clusters should be operational with one broker taken down for maintenance, so if resize is reasonably quick there is no reason to wait for online resize. I'll try as soon as we get our hands on a 1.11 cluster. We've failed to do resize in 1.9.

[solsson]

#191 includes the volume expansion setting, but not the volume size change in Kafka. I don't think we should have that until volume expansion is mainstream. Or maybe we do, with the same rationale as d088b25?

[solsson]

Actually #191 does include the changes to default volume size. Ok then.