Cluster upgrade procedure

[ant31]

Propose an upgrade procedure of an existing cluster at several levels

• Kubernetes core components
• Etcd
• Docker
• Network overlays

If it helps/simplify, we can assume that the original cluster was deploy with kubespray.

[Smana]

@mattymo @bogdando any thoughts about that ?
My recent upgrade tests it was smooth but maybe the only task required is to clean the etcd storage

[mattymo]

Why do you have to clean the ETCD storage? If anything, just back it up

[pcm32]

I have recently updated a k8s cluster running on Ubuntu 15.10 which I provisioned roughly a month ago with kubespray:

Procedure (on the machine from which I had previously deployed the k8s cluster):

sudo pip install kargo --upgrade
cd ~/.kargo
git pull
kargo deploy ... 

It failed initially because cloud_provider variable wasn't set, wasn't required last time I tried it a few weeks ago.

TASK [dnsmasq : ensure dnsmasq.d-available directory exists] *******************
ok: [k8s-inferno-4dudtp]
ok: [k8s-inferno-7by0q1]
ok: [k8s-inferno-814w19]
ok: [k8s-inferno-sv9kiu]

TASK [dnsmasq : Write dnsmasq configuration] ***********************************
fatal: [k8s-inferno-4dudtp]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'cloud_provider' is undefined"}
fatal: [k8s-inferno-7by0q1]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'cloud_provider' is undefined"}
fatal: [k8s-inferno-814w19]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'cloud_provider' is undefined"}
fatal: [k8s-inferno-sv9kiu]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'cloud_provider' is undefined"}

I set it to openstack on ~/.kargo/inventory/group_vars/all.yml, and then it failed again claiming that openstack_user and other variables were not set.

TASK [kubernetes/preinstall : check cloud_provider value] **********************
skipping: [k8s-inferno-4dudtp]
skipping: [k8s-inferno-7by0q1]
skipping: [k8s-inferno-814w19]
skipping: [k8s-inferno-sv9kiu]

TASK [kubernetes/preinstall : check openstack_auth_url value] ******************
fatal: [k8s-inferno-4dudtp]: FAILED! => {"changed": false, "failed": true, "msg": "openstack_auth_url is missing"}
fatal: [k8s-inferno-7by0q1]: FAILED! => {"changed": false, "failed": true, "msg": "openstack_auth_url is missing"}
fatal: [k8s-inferno-814w19]: FAILED! => {"changed": false, "failed": true, "msg": "openstack_auth_url is missing"}
fatal: [k8s-inferno-sv9kiu]: FAILED! => {"changed": false, "failed": true, "msg": "openstack_auth_url is missing"}

I sourced my openstack rc file, and then it got past that. It then failed again due to:

TASK [kubernetes-apps/ansible : Kubernetes Apps | Start Resources] *************
failed: [k8s-inferno-4dudtp] (item={u'src': u'/home/ubuntu/.ansible/tmp/ansible-tmp-1476656950.99-54642765919618/source', u'md5sum': u'5079d8e22ebe160b4a7689d3185af6df', u'group': u'root', u'uid': 0, u'dest': u'/etc/kubernetes/kubedns-rc.yml', u'checksum': u'cbf81992b0009b39ac7e1611d574517b45e12a3f', '_ansible_item_result': True, u'owner': u'root', '_ansible_no_log': False, 'item': {u'type': u'rc', u'file': u'kubedns-rc.yml'}, u'state': u'file', u'gid': 0, u'mode': u'0644', 'changed': True, 'invocation': {u'module_args': {u'src': u'/home/ubuntu/.ansible/tmp/ansible-tmp-1476656950.99-54642765919618/source', u'directory_mode': None, u'force': True, u'remote_src': None, u'dest': u'/etc/kubernetes/kubedns-rc.yml', u'selevel': None, u'seuser': None, u'serole': None, u'follow': True, u'content': None, u'setype': None, u'original_basename': u'kubedns-rc.yml', u'delimiter': None, u'mode': None, u'regexp': None, u'owner': None, u'group': None, u'validate': None, u'backup': False}}, u'size': 3123}) => {"failed": true, "item": {"changed": true, "checksum": "cbf81992b0009b39ac7e1611d574517b45e12a3f", "dest": "/etc/kubernetes/kubedns-rc.yml", "gid": 0, "group": "root", "invocation": {"module_args": {"backup": false, "content": null, "delimiter": null, "dest": "/etc/kubernetes/kubedns-rc.yml", "directory_mode": null, "follow": true, "force": true, "group": null, "mode": null, "original_basename": "kubedns-rc.yml", "owner": null, "regexp": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1476656950.99-54642765919618/source", "validate": null}}, "item": {"file": "kubedns-rc.yml", "type": "rc"}, "md5sum": "5079d8e22ebe160b4a7689d3185af6df", "mode": "0644", "owner": "root", "size": 3123, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1476656950.99-54642765919618/source", "state": "file", "uid": 0}, "msg": "error running kubectl (/usr/local/bin/kubectl --namespace=kube-system replace --force --filename=/etc/kubernetes/kubedns-rc.yml) command (rc=1): replicationcontroller \"kubedns\" deleted\n"}
ok: [k8s-inferno-4dudtp] => (item={u'changed': False, u'group': u'root', u'uid': 0, '_ansible_item_result': True, u'owner': u'root', 'item': {u'type': u'svc', u'file': u'kubedns-svc.yml'}, u'state': u'file', u'gid': 0, u'mode': u'0644', u'path': u'/etc/kubernetes/kubedns-svc.yml', 'invocation': {u'module_args': {u'directory_mode': None, u'force': False, u'remote_src': None, u'path': u'/etc/kubernetes/kubedns-svc.yml', u'owner': None, u'follow': True, u'group': None, u'state': None, u'content': None, u'serole': None, u'diff_peek': None, u'setype': None, u'dest': u'/etc/kubernetes/kubedns-svc.yml', u'selevel': None, u'original_basename': u'kubedns-svc.yml', u'regexp': None, u'validate': None, u'src': None, u'seuser': None, u'recurse': False, u'delimiter': None, u'mode': None, u'backup': None}}, u'diff': {u'after': {u'path': u'/etc/kubernetes/kubedns-svc.yml'}, u'before': {u'path': u'/etc/kubernetes/kubedns-svc.yml'}}, u'size': 353, '_ansible_no_log': False})

Re-running kargo deploy ... again seemed to fix it for some reason. Cluster seems to be operating fine now with hyperkube 1.4 👍 .

This was done on a live Kubernetes cluster running on top of Ubuntu 15.10, on OpenStack.