Setting WORKLOAD_CERT_TTL has no effect in Istio 1.5.0

[dahutu]

Bug description
Setting WORKLOAD_CERT_TTL has no effect in Istio 1.5.0

Expected behavior
After setting WORKLOAD_CERT_TTL environment variable for istiod and restarting it, the new WORKLOAD_CERT_TTL should reflect new cert validity

Steps to reproduce the bug
kubectl get deployment istiod -n istio-system -o yaml | tee ~/istiod.yaml

Update ~/istiod.yaml and add

        - name: WORKLOAD_CERT_TTL
          value: "10m"

To istiod environment variables section
kubectl apply -f ~/istiod.yaml

Deploy the httpbin and sleep example applications.

 kubectl create ns foo
 kubectl apply -f <(istioctl kube-inject -f samples/httpbin/httpbin.yaml) -n foo
 kubectl apply -f <(istioctl kube-inject -f samples/sleep/sleep.yaml) -n foo

Validation:

wenzhong-tang:istio-current wenzhong_tang$ kubectl exec $(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name}) -c istio-proxy -n foo -- openssl s_client -connect httpbin.foo:8000 | openssl x509 -text -noout | grep Validity -A 2
depth=2 C = US, ST = California, L = Sunnyvale, O = Istio, OU = Test, CN = Root CA, emailAddress = testrootca@istio.io
verify error:num=19:self signed certificate in certificate chain
DONE
        Validity
            Not Before: Mar  5 20:48:58 2020 GMT
            Not After : Mar  6 20:48:58 2020 GMT

The reason why this is the case is because WORKLOAD_CERT_TTL only kicks in when requested lifetime <= 0. Istio-proxy’s SDS agent defaults to 24 hours when setting the cert lifetime. Therefore, workload cert TTL is always 24 hours regardless of what value WORKLOAD_CERT_TTL has.

We currently workaround this issue by manually updating istio-sidecar-injector and update SECRET_TTL
kubectl edit cm istio-sidecar-injector -n istio-system
Then restart istiod and all pods. However we would like a central place to set it, ideally during installation.

Proposal:
Expose setting SECRET_TTL in istio-sidecar-injector config map as an installation option. Allow operators to configure workload cert lifetime.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

wenzhong-tang:~ wenzhong$ istioctl version --remote
client version: 1.5.0
control plane version: 1.5.0
data plane version: 1.5.0 (4 proxies)

We use kubernetes 1.13.

How was Istio installed?

istioctl manifest generate > ~/manifest-1_5.yaml
kubectl apply -f ~/manifest-1_5.yaml 

Environment where bug was observed (cloud vendor, OS, etc)
Kubernetes on AWS

[lei-tang]

#22072 is created to address this issue, which exposes configuring SECRET_TTL as an installation option to allow operators to configure workload cert lifetime.

[myidpt]

istioctl manifest apply --set meshConfig.defaultConfig.proxyMetadata.SECRET_TTL=10
This sets the TTL on the client side.

[dahutu]

This doesn't work. I got:
Error: failed to apply manifests: failed to generate tree from the set overlay, error: bad path=value: meshConfig.defaultConfig.proxyMetadata.SECRET_TTL=10

[myidpt]

Sorry, I missed the "m" in the end:
istioctl manifest apply --set meshConfig.defaultConfig.proxyMetadata.SECRET_TTL=10m