Helm deploy fails on pre-install hooks (AKS only)

[msiegenthaler]

Our helm deployments fail to install on the AKS cluster. The same charts work fine on other clusters including the ACS cluster.

Reproduction:

[ms@ ~] helm install --name notebook stable/tensorflow-notebook
Error: watch closed before Until timeout

(I chose the tensorflow-notebook chart for the reproduction because it's not huge and easily available. The same thing also happens with other charts)

Tiller log:

+ tiller-deploy-7ccf99cd64-5745j › tiller
tiller-deploy-7ccf99cd64-5745j tiller [main] 2018/06/20 14:35:43 Starting Tiller v2.9.1 (tls=false)
tiller-deploy-7ccf99cd64-5745j tiller [main] 2018/06/20 14:35:43 GRPC listening on :44134
tiller-deploy-7ccf99cd64-5745j tiller [main] 2018/06/20 14:35:43 Probes listening on :44135
tiller-deploy-7ccf99cd64-5745j tiller [main] 2018/06/20 14:35:43 Storage driver is ConfigMap
tiller-deploy-7ccf99cd64-5745j tiller [main] 2018/06/20 14:35:43 Max history per release is 0
tiller-deploy-7ccf99cd64-5745j tiller [tiller] 2018/06/20 14:35:53 preparing install for notebook
tiller-deploy-7ccf99cd64-5745j tiller [storage] 2018/06/20 14:35:53 getting release history for "notebook"
tiller-deploy-7ccf99cd64-5745j tiller [tiller] 2018/06/20 14:35:53 rendering tensorflow-notebook chart using values
tiller-deploy-7ccf99cd64-5745j tiller [tiller] 2018/06/20 14:35:54 performing install for notebook
tiller-deploy-7ccf99cd64-5745j tiller [tiller] 2018/06/20 14:35:54 executing 1 pre-install hooks for notebook
tiller-deploy-7ccf99cd64-5745j tiller [kube] 2018/06/20 14:35:54 building resources from manifest
tiller-deploy-7ccf99cd64-5745j tiller [kube] 2018/06/20 14:35:54 creating 1 resource(s)
tiller-deploy-7ccf99cd64-5745j tiller [kube] 2018/06/20 14:36:54 Watching for changes to Secret notebook-tensorflow-notebook with timeout of 5m0s
tiller-deploy-7ccf99cd64-5745j tiller [tiller] 2018/06/20 14:36:54 warning: Release notebook pre-install tensorflow-notebook/templates/secrets.yaml could not complete: watch closed before Until timeout
tiller-deploy-7ccf99cd64-5745j tiller [tiller] 2018/06/20 14:36:54 failed install perform step: watch closed before Until timeout

The secret did get created during this:

[ms@ ~] kubectl get secret
NAME                           TYPE                                  DATA      AGE
default-token-vvtqq            kubernetes.io/service-account-token   3         3h
notebook-tensorflow-notebook   Opaque                                1         6m

We reproduced the issue on three separate AKS clusters, all on kubernetes 1.9.6 in west and north europe. We tested with helm 2.7.0, 2.9.0, 2.9.1.
As said above, the same works without issues in our ACS cluster and on multiple terraform based ones (Kubernetes versions 1.7.7 (ACS), 1.8.5, 1.8.4).

Charts without a pre-install hook (or maybe other hooks, didn't isolate that) deploy without any issue on the AKS cluster. The cluster appears to be fine, calls via kubectl work, kubectl port-forward also works, helm list works.

[m1o1]

Same issue (referenced above: istio/istio#6301) while helm installing Istio. In my case it was failing on post-install hooks. Could this be related to AdmissionControllers?

[twendt]

This is related to #408

We had our clusters patched yesterday and now helm pre-install hooks are working again.

[m1o1]

Just tried on a new centralus cluster and it still fails

[rite2nikhil]

All (in all regions) new cluster creates should be patched with the fix. thanks for your patience, please report if you still see issues. Existing clusters will eventually get patched.

[m1o1]

I just tried again with Istio on a new cluster in eastus with rbac enabled, and I'm still seeing this issue. Here are the relevant logs for the tiller:

[tiller] 2018/06/22 12:38:30 executing 2 post-install hooks for istio
[kube] 2018/06/22 12:38:30 building resources from manifest
[kube] 2018/06/22 12:38:30 creating 1 resource(s)
[kube] 2018/06/22 12:38:31 Watching for changes to Job istio-cleanup-old-ca with timeout of 5m0s
[kube] 2018/06/22 12:38:31 Add/Modify event for istio-cleanup-old-ca: ADDED
[kube] 2018/06/22 12:38:31 istio-cleanup-old-ca: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[tiller] 2018/06/22 12:39:31 warning: Release istio post-install istio/charts/security/templates/cleanup-old-ca.yaml could not complete: watch closed before Until timeout
[tiller] 2018/06/22 12:39:31 warning: Release "istio" failed post-install: watch closed before Until timeout
[storage] 2018/06/22 12:39:31 updating release "istio.v1"
[tiller] 2018/06/22 12:39:31 failed install perform step: watch closed before Until timeout

Edit* Just tried installing the tensorflow notebook, which succeeded. Strange that Istio still fails with this.
Edit2* Same issue on centralus

[ChingizMard]

I happen to have the similar issue on AKS with post-install hooks but while installing cockroachdb. Seems like an exactly same issue as @andrew-dinunzio is having. Here are the logs:

[tiller] 2018/06/22 14:50:34 executing 2 post-install hooks for cockroachdb
[kube] 2018/06/22 14:50:34 building resources from manifest
[kube] 2018/06/22 14:50:34 creating 1 resource(s)
[kube] 2018/06/22 14:50:34 Watching for changes to Job cockroachdb-cockroachdb-init with timeout of 5m0s
[kube] 2018/06/22 14:50:34 Add/Modify event for cockroachdb-cockroachdb-init: ADDED
[kube] 2018/06/22 14:50:34 cockroachdb-cockroachdb-init: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
[kube] 2018/06/22 14:50:34 Add/Modify event for cockroachdb-cockroachdb-init: MODIFIED
[kube] 2018/06/22 14:50:34 cockroachdb-cockroachdb-init: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[tiller] 2018/06/22 14:51:34 warning: Release cockroachdb post-install cockroachdb/templates/cluster-init.yaml could not complete: watch closed before Until timeout
[tiller] 2018/06/22 14:51:34 warning: Release "cockroachdb" failed post-install: watch closed before Until timeout
[storage] 2018/06/22 14:51:34 updating release "cockroachdb.v1"
[tiller] 2018/06/22 14:51:34 failed install perform step: watch closed before Until timeout

[rite2nikhil]

This is caused by the idle timeout on watches, currently this is expected to be ~60s, aware of this issue, partial fix rolling out next week, working at high priority on a full fix.

Thanks for reporting the issue.

[msiegenthaler]

Works for me now after upgrading the cluster to 1.10.3

[rlucioni]

The 60-second idle timeout on watches is also preventing our post-install hooks from succeeding. Our cluster is running 1.9.6.

[m1o1]

We tested with 1.10.3 and it still fails. It's a configuration of the api-server that's responsible for this I think.

[m1o1]

There's another issue with Istio that might be related to this watch timeout issue. Looking at the logs for the mixer with kubectl logs -n istio-system istio-telemetry-54b5bf4847-l9qqt mixer, I see the following logs:

2018-06-25T10:17:50.439694Z     error   istio.io/istio/mixer/pkg/config/crd/store.go:169: Failed to watch *unstructured.Unstructured: the server was unable to return a response in the time allotted, but may still be processing the request (get kuberneteses.config.istio.io)

The container never crashes - it just repeats this error repeatedly. Does not happen on minikube.

[rite2nikhil]

We have rolled out fixes, please provide feedback if there are issues with new cluster creates

[StianOvrevage]

We had problems with 1.10.3 and the prometheus-operator chart. Will try again today on a new cluster.

prometheus-operator/prometheus-operator#1514 (comment)