Reapplying istio-demo.yml causes cert errors in injector

[laszlocph]

The oneliner kubectl apply -f istio-demo.yml is useful, but when I make a modification and reapply it, the sidecar injector stops working and throws a lot of remote error: tls: bad certificate errors.

I think yamls should be idempotent.

Any idea how to solve this?

[gyliu513]

This is more like a kubernetes issue, how did you change your YAML template? A diff would be helpful between istio-demo.yaml and your updated one.

[laszlocph]

It happens without any change.

Steps to reproduce:

curl -L https://git.io/getLatestIstio | sh -
cd istio-0.8.0
kubectl apply -f install/kubernetes/istio-demo.yaml
sleep 120
kubectl apply -f install/kubernetes/istio-demo.yaml
k logs -f istio-sidecar-injector-645c89bc64-54b9h -n istio-system

Then deploy in a namespace with istio-injection=enabled and see the logs of the injector

2018-06-07T12:48:17.351968Z     info    http: TLS handshake error from 10.xxx.133.1:37886: remote error: tls: bad certificate
2018-06-07T12:48:27.609948Z     info    http: TLS handshake error from 10.xxx.0.6:38626: remote error: tls: bad certificate
2018-06-07T12:48:48.109024Z     info    http: TLS handshake error from 10.xxx.0.6:38728: remote error: tls: bad certificate

no pods get scheduled.

[gyliu513]

/cc @ayj

[ayj]

The MutatingWebhookConfiguration in istio.yaml includes an empty caBundle. The sidecar injector pod patches the caBundle at start-up. Re-applying istio.yaml erases the patched caBundle leading to TLS handshake errors. A few options:

1. restart the injector pod to re-patch the caBundle
2. teach the injector to watch the caBundle and automatically re-patch
3. write a proper webhook installer job that intelligently applies the webhook configuration and patches the caBundle. The job would be run as part of istio.yaml, similar to the mixer's post-install job for installing CRDs.

cc @costinm @ostromart

[ghost]

just for clarity, this issue isn't unique to the demo example. We're seeing the same issue on 0.8 using the helm template install. Restarting the injector pod does resolve the problem for me.

[Rigdon]

I just hit this as well on the 0.8 release. Oddly though it took a couple injector pod restarts to patch the ca and there wasn't anything telling in the injector pod logs about it failing to patch. If it happens again I'll collect more info.

[ostromart]

#6435

[thecodejunkie]

Hi,

I believe I just experienced this with 1.3.1. I generated manifests using the Helm charts and installed Istio-init, Istio-Cni, and Istio in my cluster. After everything has been applied, the caBundle field, of the MutatingWebhookConfiguration is still blank (or not even there)

- clientConfig:
    service:
      name: istio-sidecar-injector
      namespace: istio-system
      path: /inject

after I restart the sidecar injector pod I can see that the caBundle filed it properly filled out.

As a bit of context, I install Istio-init, Istio-Cni, and Istio manifests using Anthos Config Management so I don't have any control of the order in which they are applied. Could this be a chicken/egg problem?

[thecodejunkie]

@ayj @costinm @ostromart 😄