New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I upgrade tiller on ACS? #55

Open
bacongobbler opened this Issue Aug 17, 2017 · 18 comments

Comments

Projects
None yet
8 participants
@bacongobbler
Member

bacongobbler commented Aug 17, 2017

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Every time I try to bump the tiller version up to v2.6.0 in ACS, ACS keeps preventing that by re-deploying tiller v2.5.1, which is what was provided when bootstrapping the cluster. I'd like to upgrade tiller on my cluster, but I cannot do that without switching over to acs-engine to deploy a cluster.

@JackQuincy

This comment has been minimized.

Show comment
Hide comment
@JackQuincy

JackQuincy Aug 17, 2017

Member

This appears to be because of how we are defining our tiller addon. We have it marked as a cluster service. Which means Kuberentes doesn't want to allow updates to it through the kubernetes api. Instead it wants the updates to happen on the file in the master. We could change to using addonmanager.kubernetes.io/mode=EnsureExists . @anhowe @seanknox @jackfrancis thoughts?

relevant Kubernetes documentation
https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/README.md

Member

JackQuincy commented Aug 17, 2017

This appears to be because of how we are defining our tiller addon. We have it marked as a cluster service. Which means Kuberentes doesn't want to allow updates to it through the kubernetes api. Instead it wants the updates to happen on the file in the master. We could change to using addonmanager.kubernetes.io/mode=EnsureExists . @anhowe @seanknox @jackfrancis thoughts?

relevant Kubernetes documentation
https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/README.md

@seanknox

This comment has been minimized.

Show comment
Hide comment
@seanknox

seanknox Aug 17, 2017

Member

addonmanager.kubernetes.io/mode=EnsureExists appears to be right call here. Agree @lachie83?

Member

seanknox commented Aug 17, 2017

addonmanager.kubernetes.io/mode=EnsureExists appears to be right call here. Agree @lachie83?

@JackQuincy

This comment has been minimized.

Show comment
Hide comment
@JackQuincy

JackQuincy Aug 17, 2017

Member

Also what do we think about doing this for dashboard, heapster, and the storage classes addons? Do we want them static or do we want customers to be able to modify them through the api server

Member

JackQuincy commented Aug 17, 2017

Also what do we think about doing this for dashboard, heapster, and the storage classes addons? Do we want them static or do we want customers to be able to modify them through the api server

@seanknox

This comment has been minimized.

Show comment
Hide comment
@seanknox

seanknox Aug 17, 2017

Member

Good call for dashboard and heapster, though my hunch without thinking through much is that we should err on the more prescriptive end with storage classes.

Member

seanknox commented Aug 17, 2017

Good call for dashboard and heapster, though my hunch without thinking through much is that we should err on the more prescriptive end with storage classes.

@JackQuincy

This comment has been minimized.

Show comment
Hide comment
@JackQuincy

JackQuincy Aug 17, 2017

Member

I think I agree on not letting people shoot themselves in the foot with storage classes. Mostly I was wanting to look at all the extensions and decide if we want to update them while doing this.

Member

JackQuincy commented Aug 17, 2017

I think I agree on not letting people shoot themselves in the foot with storage classes. Mostly I was wanting to look at all the extensions and decide if we want to update them while doing this.

@JackQuincy

This comment has been minimized.

Show comment
Hide comment
@JackQuincy

JackQuincy Aug 17, 2017

Member

@bacongobbler if it wasn't clear there is a work around by updating the add-on file for tiller, /etc/kubernetes/addons/kube-tiller-deployment.yaml, to be the newer spec for tiller on each master.

Member

JackQuincy commented Aug 17, 2017

@bacongobbler if it wasn't clear there is a work around by updating the add-on file for tiller, /etc/kubernetes/addons/kube-tiller-deployment.yaml, to be the newer spec for tiller on each master.

@lachie83

This comment has been minimized.

Show comment
Hide comment
@lachie83

lachie83 Aug 17, 2017

Member

Confirming everything @JackQuincy @seanknox have said.

Member

lachie83 commented Aug 17, 2017

Confirming everything @JackQuincy @seanknox have said.

@bacongobbler

This comment has been minimized.

Show comment
Hide comment
@bacongobbler

bacongobbler Aug 18, 2017

Member

I was able to work around this by deploying a hacked up acs-engine cluster with Tiller v2.6.0 but I'll keep that in mind for next time. Thanks @JackQuincy @seanknox @lachie83 for your help!

Member

bacongobbler commented Aug 18, 2017

I was able to work around this by deploying a hacked up acs-engine cluster with Tiller v2.6.0 but I'll keep that in mind for next time. Thanks @JackQuincy @seanknox @lachie83 for your help!

@berndverst

This comment has been minimized.

Show comment
Hide comment
@berndverst

berndverst Aug 23, 2017

Member

Documenting my single command workaround based on comments above (make sure to replace the host with your ACS master node)

ssh azureuser@draftdeploymgmt.westus.cloudapp.azure.com sudo sed -i s/'2.5.1'/'2.6.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade
Member

berndverst commented Aug 23, 2017

Documenting my single command workaround based on comments above (make sure to replace the host with your ACS master node)

ssh azureuser@draftdeploymgmt.westus.cloudapp.azure.com sudo sed -i s/'2.5.1'/'2.6.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade
@bacongobbler

This comment has been minimized.

Show comment
Hide comment
@bacongobbler

bacongobbler Aug 23, 2017

Member

@berndverst I tried your example and it worked, however I would suggest restarting the kube-addon-manager pod by deleting it from the kube-system namespace, e.g.

$ kubectl -n kube-system delete po kube-addon-manager-k8s-master-a637ed6-0

This will force a new addon-manager pod to boot and reconcile all the resources managed by the addon manager... Including tiller. This will force a fresh tiller v2.6.0 instance to be deployed.

Member

bacongobbler commented Aug 23, 2017

@berndverst I tried your example and it worked, however I would suggest restarting the kube-addon-manager pod by deleting it from the kube-system namespace, e.g.

$ kubectl -n kube-system delete po kube-addon-manager-k8s-master-a637ed6-0

This will force a new addon-manager pod to boot and reconcile all the resources managed by the addon manager... Including tiller. This will force a fresh tiller v2.6.0 instance to be deployed.

@seanknox

This comment has been minimized.

Show comment
Hide comment
@seanknox

seanknox Aug 23, 2017

Member

JFYI we have the long-term fix for this slated to go into acs-engine soon: Azure/acs-engine#1319

Member

seanknox commented Aug 23, 2017

JFYI we have the long-term fix for this slated to go into acs-engine soon: Azure/acs-engine#1319

@stuartleeks

This comment has been minimized.

Show comment
Hide comment
@stuartleeks

stuartleeks Aug 25, 2017

What is the timeline for this being fixed in ACS?

stuartleeks commented Aug 25, 2017

What is the timeline for this being fixed in ACS?

@JackQuincy

This comment has been minimized.

Show comment
Hide comment
@JackQuincy

JackQuincy Aug 25, 2017

Member

It will be in the v2 regions in a week or two if it isn't already. In the v1 regions it will be next month.

Member

JackQuincy commented Aug 25, 2017

It will be in the v2 regions in a week or two if it isn't already. In the v1 regions it will be next month.

@subho007

This comment has been minimized.

Show comment
Hide comment
@subho007

subho007 Sep 14, 2017

Any updates on this?

subho007 commented Sep 14, 2017

Any updates on this?

@JackQuincy

This comment has been minimized.

Show comment
Hide comment
@JackQuincy

JackQuincy Sep 15, 2017

Member

Definetly in ACSv2. V1 is still a ways out we have had a deployment having other issues that is delaying this some.

Member

JackQuincy commented Sep 15, 2017

Definetly in ACSv2. V1 is still a ways out we have had a deployment having other issues that is delaying this some.

@lrills

This comment has been minimized.

Show comment
Hide comment
@lrills

lrills Oct 31, 2017

Any update on this? Cluster in Southeast Asia still can't upgrade with helm init --upgade now.

lrills commented Oct 31, 2017

Any update on this? Cluster in Southeast Asia still can't upgrade with helm init --upgade now.

@berndverst

This comment has been minimized.

Show comment
Hide comment
@berndverst

berndverst Nov 5, 2017

Member

@lrills I manually had to update the add on definition on the Kubernetes master again as a workaround. Like so:

ssh azureuser@MYK8SMASTERHOST.westus.cloudapp.azure.com sudo sed -i s/'2.6.0'/'2.7.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade

No other method has reliably worked for me.

Member

berndverst commented Nov 5, 2017

@lrills I manually had to update the add on definition on the Kubernetes master again as a workaround. Like so:

ssh azureuser@MYK8SMASTERHOST.westus.cloudapp.azure.com sudo sed -i s/'2.6.0'/'2.7.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade

No other method has reliably worked for me.

@bacongobbler

This comment has been minimized.

Show comment
Hide comment
@bacongobbler

bacongobbler Nov 5, 2017

Member

I’m going to reopen this for visibility. I’m still making weekly references to this ticket in #helm-users. :)

Member

bacongobbler commented Nov 5, 2017

I’m going to reopen this for visibility. I’m still making weekly references to this ticket in #helm-users. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment