Upgrade path for the zookeeper persistence issue

[solsson]

#227 for a v4.3.1 release, fixing #89, with an upgrade path to migrate your zookeeper state. Recovery from snapshots is an alternative, but I haven't looked into that.

v5.0.0 hasn't been out long enough for anyone to run anything important on, I suppose. Thus we should focus on how to migrate a 4.3.0 installation, that might have been running since v2.0.0. After this upgrade I think that it'll be a straightforward kubectl replace to go to v5.0.1.

[solsson]

Edit: Got an idea for a less risky upgrade flow that doesn't require downtime and doesn't involve filesystem operations on running pods. See new comment below.

First step of an upgrade is to take a healthy cluster and replace the ephemeral zoo with a updated variant. No need to migrate data.

kubectl replace -f zookeeper/10zookeeper-config.yml
kubectl delete -f zookeeper/51zoo.yml && kubectl apply -f zookeeper/51zoo.yml

Now take a pause and make sure your cluster is still working. When everything is stable do:

kubectl -n kafka exec pzoo-2 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl -n kafka exec pzoo-1 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl -n kafka exec pzoo-0 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl delete -f zookeeper/50pzoo.yml && kubectl apply -f zookeeper/50pzoo.yml

Check for example kubectl -n kafka exec pzoo-0 -- ls -lR /var/lib/zookeeper before and after. This process obviously comes with no guarantees. A file-system level backup of /var/lib/zookeeper from all 5 pods is recommended.

[solsson]

Not recommended for production yet, but I upgraded to v5.0.1 after the above, like so:

kubectl apply -f kafka/
kubectl delete -f kafka/50kafka.yml && kubectl apply -f kafka/50kafka.yml
# pause here to await readiness
kubectl apply -f zookeeper/
kubectl delete -f zookeeper/51zoo.yml && kubectl apply -f zookeeper/51zoo.yml
# pause here to await radiness
kubectl delete -f zookeeper/50pzoo.yml && kubectl apply -f zookeeper/50pzoo.yml

[solsson]

An upgrade flow that can avoid downtime, I think, is to make use of the fact that zoo has ephemeral storage and kubectl apply is allowed for the configmap and (to quote the error message from apply or replace on the statefulset) "updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden."

Thus we can scale up zoo to form a majority, by applying https://github.com/Yolean/kubernetes-kafka/compare/v4.3.0...4.3.1-upgrade-preparations?expand=1` and then allow ourselves to ignore the data lost when restarting pzoo pods (backup recommended as always).

Note that for the configmap to have effect you must delete the old pods one by one after the two new pods have become ready. The suggested order is zoo-1, zoo-0 (and now is a good time to check logs of zoo pods), pzoo-2, pzoo-1, pzoo-0.

After that comes the risky part: you can switch to the v4.3.1 release, apply 0ed261f there, apply the configmap again and then delete+apply as above the pzoo statefulset. Between delete and apply you might want to delete the pzoo volumes to avoid an old snapshots dir in your volumes.

Finally revert the +2 scale patch, apply the configmap, delete the pzoo pods one by one to give them the majority, delete+apply zoo.

As you can see ideas in #225 are most welcome.

Note that pzoo is more difficult to scale, because you'd have to edit the ID_OFFSET and restart zoo pods as well, and at some point in this process you risk id collisions.

[solsson]

I ran the above upgrade flow now using the commands below. That this is not a script: you need to run the commands one by one and make sure everything is up and healthy inbetween:

# not a script, everything must become ready again after every step
git checkout 4.3.1-upgrade-preparations
kubectl apply -f zookeeper/
kubectl -n kafka delete pod zoo-1
kubectl -n kafka delete pod zoo-0
kubectl -n kafka delete pod pzoo-2
kubectl -n kafka delete pod pzoo-1
kubectl -n kafka delete pod pzoo-0
git checkout v4.3.1
git cherry-pick 0ed261f
kubectl apply -f zookeeper/10zookeeper-config.yml
kubectl delete -f zookeeper/50pzoo.yml
kubectl apply -f zookeeper/50pzoo.yml
git checkout v4.3.1
kubectl apply -f zookeeper/10zookeeper-config.yml 
kubectl -n kafka delete pod pzoo-2
kubectl -n kafka delete pod pzoo-1
kubectl -n kafka delete pod pzoo-0
kubectl delete -f zookeeper/51zoo.yml
kubectl apply -f zookeeper/51zoo.yml

I had tests and eventrouter running, with no interruptions. I also tested deletion of kafka pods after the delete steps and everything restarted normally. I deleted two of the three pzoo volume claims, which shows that deletion is optional.

[solsson]

I think it's actually possible to upgrade using apply only, no deletion of statefulset. Probably without the need to scale up zoo. The reason I had to delete was that I was working with #227 i.e. version 5.0+ when this issue was discovered.

Edit: Which means #228 (comment), with other git tag names, is our upgrade flow to v5.0.3 :)

[solsson]

Confirmed. I've upgraded without scaling:

Directly from v4.3.1, no additional patch:

kubectl apply -f zookeeper/10zookeeper-config.yml
kubectl apply -f zookeeper/51zoo.yml
kubectl apply -f zookeeper/50pzoo.yml
# ... pause, wait for readiness ...
kubectl -n kafka delete pod zoo-1
# ... pause, wait for readiness ...
kubectl -n kafka delete pod zoo-0
# ... pause, wait for readiness ...
kubectl -n kafka exec pzoo-2 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/' && kubectl -n kafka delete pod pzoo-2
# ... pause, wait for readiness ...
kubectl -n kafka exec pzoo-1 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/' && kubectl -n kafka delete pod pzoo-1
# ... pause, wait for readiness ...
kubectl -n kafka exec pzoo-0 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/' && kubectl -n kafka delete pod pzoo-0