Add initial docs for how to rotate a CA keypair

docs/rotate-secrets.md

@@ -0,0 +1,46 @@
+# How to rotate all secrets / credentials
+
+This is a disruptive procedure.
+
+Delete all secrets & keypairs that kops is holding:
+
+```
+kops get secrets  | grep ^Secret | awk '{print $2}' | xargs -I {} kops delete secret secret {}
+
+kops get secrets  | grep ^Keypair | awk '{print $2}' | xargs -I {} kops delete secret keypair {}
+```
+
+Now run `kops update cluster` and `kops update cluster --yes` to regenerate the secrets & keypairs.
+
+We need to reboot every node (using a rolling-update).  We have to use `--cloudonly` because our keypair no longer matches.
+We set the interval small because nodes will stop trusting each other during the process, so there is no point in going slowly.
+
+`kops rolling-update cluster --cloudonly --master-interval=10s --node-interval=10s --force --yes`
+
+Re-export kubecfg with new settings:
+
+`kops export kubecfg`
+
+Now the service account tokens will need to be regenerated inside the cluster:
+
+`kops toolbox dump` and find a master IP
+
+Then `ssh admin@${IP}` and run this to delete all the service account tokens:
+
+```
+# Delete all service account tokens in all namespaces
+NS=`kubectl get namespaces -o 'jsonpath={.items[*].metadata.name}'`
+for i in ${NS}; do kubectl get secrets --namespace=${i} --no-headers | grep "kubernetes.io/service-account-token" | awk '{print $1}' | xargs -I {} kubectl delete secret --namespace=$i {}; done
+
+# Allow for new secrets to be created
+sleep 60
+
+# Bounce pods that we know use service account tokens - you will likely have to bounce more
+kubectl delete pods -lk8s-app=dns-controller --namespace=kube-system
+kubectl delete pods -lk8s-app=kube-dns --namespace=kube-system
+kubectl delete pods -lk8s-app=kube-dns-autoscaler --namespace=kube-system
+pkill -f kube-controller-manager
+```
+
+
+