EKS API server downtime may corrupt "istio-sidecar-injector"?

[mak-1-sim]

That bug may be a replication of #17718 or #20478 but they have closed and nothing from them helps with my issue.

I faced the issue with the istio-sidecar-injector on EKS. Before that issue, I got notification from Alertmanager that the EKS API server has short-time downtime.

k8s describe of failed ReplicaSet :

Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  14m (x19 over 36m)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate has expired or is not yet valid

Galley logs:

2020-06-03T14:22:53.621936Z    info    validation    istio-galley validatingwebhookconfiguration unchanged, no update needed
2020-06-03T14:22:53.645761Z    info    validation    istio-galley validatingwebhookconfiguration unchanged, no update needed
2020-06-03T14:23:14.085318Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: NEW (ResourceSource), supported collections: string{"istio/rbac/v1alpha1/rbacconfigs", "istio/config/v1alpha2/legacy/deniers", "istio/config/v1alpha2/legacy/kuberneteses", "istio/rbac/v1alpha1/clusterrbacconfigs", "istio/config/v1alpha2/legacy/prometheuses", "istio/config/v1alpha2/legacy/logentries", "istio/authentication/v1alpha1/meshpolicies", "istio/policy/v1beta1/attributemanifests", "istio/config/v1alpha2/legacy/rbacs", "istio/policy/v1beta1/handlers", "istio/authentication/v1alpha1/policies", "istio/networking/v1alpha3/gateways", "istio/config/v1alpha2/legacy/circonuses", "istio/config/v1alpha2/legacy/bypasses", "istio/config/v1alpha2/legacy/zipkins", "istio/networking/v1alpha3/envoyfilters", "istio/policy/v1beta1/instances", "istio/config/v1alpha2/legacy/listentries", "istio/config/v1alpha2/legacy/apikeys", "istio/networking/v1alpha3/synthetic/serviceentries", "istio/config/v1alpha2/legacy/cloudwatches", "k8s/extensions/v1beta1/ingresses", "istio/config/v1alpha2/legacy/checknothings", "istio/config/v1alpha2/templates", "istio/networking/v1alpha3/destinationrules", "istio/networking/v1alpha3/virtualservices", "istio/mixer/v1/config/client/quotaspecbindings", "istio/config/v1alpha2/legacy/statsds", "istio/config/v1alpha2/legacy/quotas", "istio/policy/v1beta1/rules", "istio/config/v1alpha2/legacy/stackdrivers", "istio/rbac/v1alpha1/serviceroles", "istio/rbac/v1alpha1/servicerolebindings", "istio/config/v1alpha2/legacy/noops", "istio/mesh/v1alpha1/MeshConfig", "istio/config/v1alpha2/legacy/solarwindses", "istio/config/v1alpha2/httpapispecs", "istio/config/v1alpha2/legacy/kubernetesenvs", "istio/config/v1alpha2/legacy/redisquotas", "istio/config/v1alpha2/httpapispecbindings", "istio/config/v1alpha2/legacy/opas", "istio/config/v1alpha2/legacy/dogstatsds", "istio/networking/v1alpha3/serviceentries", "istio/config/v1alpha2/legacy/fluentds", "istio/config/v1alpha2/legacy/edges", "istio/mixer/v1/config/client/quotaspecs", "istio/config/v1alpha2/legacy/listcheckers", "istio/config/v1alpha2/adapters", "istio/config/v1alpha2/legacy/tracespans", "istio/config/v1alpha2/legacy/stdios", "istio/config/v1alpha2/legacy/memquotas", "istio/config/v1alpha2/legacy/signalfxs", "istio/config/v1alpha2/legacy/authorizations", "istio/config/v1alpha2/legacy/reportnothings", "istio/config/v1alpha2/legacy/metrics", "istio/networking/v1alpha3/sidecars"}
2020-06-03T14:23:14.085413Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/kuberneteses
2020-06-03T14:23:14.085441Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/logentries
2020-06-03T14:23:14.085466Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/authorizations
2020-06-03T14:23:14.085478Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/adapters
2020-06-03T14:23:14.085503Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/metrics
2020-06-03T14:23:14.085521Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/policy/v1beta1/handlers
2020-06-03T14:23:14.085560Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/reportnothings
2020-06-03T14:23:14.085574Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/checknothings
2020-06-03T14:23:14.085599Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/policy/v1beta1/rules
2020-06-03T14:23:14.085615Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/templates
2020-06-03T14:23:14.085657Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/listentries
2020-06-03T14:23:14.085673Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/policy/v1beta1/attributemanifests
2020-06-03T14:23:14.085715Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/quotas
2020-06-03T14:23:14.085731Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/policy/v1beta1/instances
2020-06-03T14:23:14.085784Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/edges
2020-06-03T14:23:14.085806Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/apikeys
2020-06-03T14:23:14.085826Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1}: inc=false WATCH for istio/config/v1alpha2/legacy/tracespans
2020-06-03T14:23:14.086152Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: NEW (ResourceSource), supported collections: string{"istio/rbac/v1alpha1/rbacconfigs", "istio/config/v1alpha2/legacy/deniers", "istio/config/v1alpha2/legacy/kuberneteses", "istio/rbac/v1alpha1/clusterrbacconfigs", "istio/config/v1alpha2/legacy/prometheuses", "istio/config/v1alpha2/legacy/logentries", "istio/authentication/v1alpha1/meshpolicies", "istio/policy/v1beta1/attributemanifests", "istio/config/v1alpha2/legacy/rbacs", "istio/policy/v1beta1/handlers", "istio/authentication/v1alpha1/policies", "istio/networking/v1alpha3/gateways", "istio/config/v1alpha2/legacy/circonuses", "istio/config/v1alpha2/legacy/bypasses", "istio/config/v1alpha2/legacy/zipkins", "istio/networking/v1alpha3/envoyfilters", "istio/policy/v1beta1/instances", "istio/config/v1alpha2/legacy/listentries", "istio/config/v1alpha2/legacy/apikeys", "istio/networking/v1alpha3/synthetic/serviceentries", "istio/config/v1alpha2/legacy/cloudwatches", "k8s/extensions/v1beta1/ingresses", "istio/config/v1alpha2/legacy/checknothings", "istio/config/v1alpha2/templates", "istio/networking/v1alpha3/destinationrules", "istio/networking/v1alpha3/virtualservices", "istio/mixer/v1/config/client/quotaspecbindings", "istio/config/v1alpha2/legacy/statsds", "istio/config/v1alpha2/legacy/quotas", "istio/policy/v1beta1/rules", "istio/config/v1alpha2/legacy/stackdrivers", "istio/rbac/v1alpha1/serviceroles", "istio/rbac/v1alpha1/servicerolebindings", "istio/config/v1alpha2/legacy/noops", "istio/mesh/v1alpha1/MeshConfig", "istio/config/v1alpha2/legacy/solarwindses", "istio/config/v1alpha2/httpapispecs", "istio/config/v1alpha2/legacy/kubernetesenvs", "istio/config/v1alpha2/legacy/redisquotas", "istio/config/v1alpha2/httpapispecbindings", "istio/config/v1alpha2/legacy/opas", "istio/config/v1alpha2/legacy/dogstatsds", "istio/networking/v1alpha3/serviceentries", "istio/config/v1alpha2/legacy/fluentds", "istio/config/v1alpha2/legacy/edges", "istio/mixer/v1/config/client/quotaspecs", "istio/config/v1alpha2/legacy/listcheckers", "istio/config/v1alpha2/adapters", "istio/config/v1alpha2/legacy/tracespans", "istio/config/v1alpha2/legacy/stdios", "istio/config/v1alpha2/legacy/memquotas", "istio/config/v1alpha2/legacy/signalfxs", "istio/config/v1alpha2/legacy/authorizations", "istio/config/v1alpha2/legacy/reportnothings", "istio/config/v1alpha2/legacy/metrics", "istio/networking/v1alpha3/sidecars"}
2020-06-03T14:23:14.086358Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/config/v1alpha2/httpapispecs
2020-06-03T14:23:14.086464Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/config/v1alpha2/httpapispecbindings
2020-06-03T14:23:14.087117Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3}: NEW (ResourceSource), supported collections: string{"istio/rbac/v1alpha1/rbacconfigs", "istio/config/v1alpha2/legacy/deniers", "istio/config/v1alpha2/legacy/kuberneteses", "istio/rbac/v1alpha1/clusterrbacconfigs", "istio/config/v1alpha2/legacy/prometheuses", "istio/config/v1alpha2/legacy/logentries", "istio/authentication/v1alpha1/meshpolicies", "istio/policy/v1beta1/attributemanifests", "istio/config/v1alpha2/legacy/rbacs", "istio/policy/v1beta1/handlers", "istio/authentication/v1alpha1/policies", "istio/networking/v1alpha3/gateways", "istio/config/v1alpha2/legacy/circonuses", "istio/config/v1alpha2/legacy/bypasses", "istio/config/v1alpha2/legacy/zipkins", "istio/networking/v1alpha3/envoyfilters", "istio/policy/v1beta1/instances", "istio/config/v1alpha2/legacy/listentries", "istio/config/v1alpha2/legacy/apikeys", "istio/networking/v1alpha3/synthetic/serviceentries", "istio/config/v1alpha2/legacy/cloudwatches", "k8s/extensions/v1beta1/ingresses", "istio/config/v1alpha2/legacy/checknothings", "istio/config/v1alpha2/templates", "istio/networking/v1alpha3/destinationrules", "istio/networking/v1alpha3/virtualservices", "istio/mixer/v1/config/client/quotaspecbindings", "istio/config/v1alpha2/legacy/statsds", "istio/config/v1alpha2/legacy/quotas", "istio/policy/v1beta1/rules", "istio/config/v1alpha2/legacy/stackdrivers", "istio/rbac/v1alpha1/serviceroles", "istio/rbac/v1alpha1/servicerolebindings", "istio/config/v1alpha2/legacy/noops", "istio/mesh/v1alpha1/MeshConfig", "istio/config/v1alpha2/legacy/solarwindses", "istio/config/v1alpha2/httpapispecs", "istio/config/v1alpha2/legacy/kubernetesenvs", "istio/config/v1alpha2/legacy/redisquotas", "istio/config/v1alpha2/httpapispecbindings", "istio/config/v1alpha2/legacy/opas", "istio/config/v1alpha2/legacy/dogstatsds", "istio/networking/v1alpha3/serviceentries", "istio/config/v1alpha2/legacy/fluentds", "istio/config/v1alpha2/legacy/edges", "istio/mixer/v1/config/client/quotaspecs", "istio/config/v1alpha2/legacy/listcheckers", "istio/config/v1alpha2/adapters", "istio/config/v1alpha2/legacy/tracespans", "istio/config/v1alpha2/legacy/stdios", "istio/config/v1alpha2/legacy/memquotas", "istio/config/v1alpha2/legacy/signalfxs", "istio/config/v1alpha2/legacy/authorizations", "istio/config/v1alpha2/legacy/reportnothings", "istio/config/v1alpha2/legacy/metrics", "istio/networking/v1alpha3/sidecars"}
2020-06-03T14:23:14.087441Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3}: inc=false WATCH for istio/policy/v1beta1/handlers
2020-06-03T14:23:14.087604Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/mixer/v1/config/client/quotaspecs
2020-06-03T14:23:14.087574Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3}: inc=false WATCH for istio/policy/v1beta1/instances
2020-06-03T14:23:14.087957Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/mixer/v1/config/client/quotaspecbindings
2020-06-03T14:23:14.088703Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/authentication/v1alpha1/policies
2020-06-03T14:23:14.089037Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/networking/v1alpha3/gateways
2020-06-03T14:23:14.089089Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/networking/v1alpha3/destinationrules
2020-06-03T14:23:14.089153Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/networking/v1alpha3/sidecars
2020-06-03T14:23:14.089184Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/rbac/v1alpha1/clusterrbacconfigs
2020-06-03T14:23:14.089326Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/networking/v1alpha3/virtualservices
2020-06-03T14:23:14.089361Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/networking/v1alpha3/envoyfilters
2020-06-03T14:23:14.089709Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/authentication/v1alpha1/meshpolicies
2020-06-03T14:23:14.089751Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/rbac/v1alpha1/serviceroles
2020-06-03T14:23:14.089801Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/rbac/v1alpha1/rbacconfigs
2020-06-03T14:23:14.089830Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/networking/v1alpha3/serviceentries
2020-06-03T14:23:14.089883Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2}: inc=false WATCH for istio/rbac/v1alpha1/servicerolebindings
2020-06-03T14:23:14.088171Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3}: inc=false WATCH for istio/policy/v1beta1/rules
2020-06-03T14:23:14.090483Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3}: inc=false WATCH for istio/config/v1alpha2/templates
2020-06-03T14:23:14.090558Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/kuberneteses with version="0" nonce="1" inc=false
2020-06-03T14:23:14.090621Z    info    mcp    Watch(): created watch 1 for istio/config/v1alpha2/legacy/kuberneteses from group "default", version "0"
2020-06-03T14:23:14.090678Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/logentries with version="0" nonce="2" inc=false
2020-06-03T14:23:14.090717Z    info    mcp    Watch(): created watch 2 for istio/config/v1alpha2/legacy/logentries from group "default", version "0"
2020-06-03T14:23:14.090685Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3}: inc=false WATCH for istio/config/v1alpha2/adapters
2020-06-03T14:23:14.090794Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3}: inc=false WATCH for istio/policy/v1beta1/attributemanifests
2020-06-03T14:23:14.091001Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/config/v1alpha2/httpapispecs with version="0" nonce="1" inc=false
2020-06-03T14:23:14.091037Z    info    mcp    Watch(): created watch 3 for istio/config/v1alpha2/httpapispecs from group "default", version "0"
2020-06-03T14:23:14.091070Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/config/v1alpha2/httpapispecbindings with version="0" nonce="2" inc=false
2020-06-03T14:23:14.091097Z    info    mcp    Watch(): created watch 4 for istio/config/v1alpha2/httpapispecbindings from group "default", version "0"
2020-06-03T14:23:14.091129Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/mixer/v1/config/client/quotaspecs with version="0" nonce="3" inc=false
2020-06-03T14:23:14.091155Z    info    mcp    Watch(): created watch 5 for istio/mixer/v1/config/client/quotaspecs from group "default", version "0"
2020-06-03T14:23:14.091301Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/authorizations with version="0" nonce="3" inc=false
2020-06-03T14:23:14.091374Z    info    mcp    Watch(): created watch 6 for istio/config/v1alpha2/legacy/authorizations from group "default", version "0"
2020-06-03T14:23:14.091435Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/adapters with version="0" nonce="4" inc=false
2020-06-03T14:23:14.091376Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/mixer/v1/config/client/quotaspecbindings with version="0" nonce="4" inc=false
2020-06-03T14:23:14.091480Z    info    mcp    Watch(): created watch 7 for istio/mixer/v1/config/client/quotaspecbindings from group "default", version "0"
2020-06-03T14:23:14.091499Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/authentication/v1alpha1/policies with version="0" nonce="5" inc=false
2020-06-03T14:23:14.091541Z    info    mcp    Watch(): created watch 8 for istio/config/v1alpha2/adapters from group "default", version "0"
2020-06-03T14:23:14.091594Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/metrics with version="0" nonce="5" inc=false
2020-06-03T14:23:14.091625Z    info    mcp    Watch(): created watch 9 for istio/config/v1alpha2/legacy/metrics from group "default", version "0"
2020-06-03T14:23:14.091715Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/policy/v1beta1/handlers with version="12" nonce="6" inc=false
2020-06-03T14:23:14.091792Z    info    mcp    Watch(): created watch 10 for istio/policy/v1beta1/handlers from group "default", version "12"
2020-06-03T14:23:14.091850Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/reportnothings with version="0" nonce="7" inc=false
2020-06-03T14:23:14.091882Z    info    mcp    Watch(): created watch 11 for istio/config/v1alpha2/legacy/reportnothings from group "default", version "0"
2020-06-03T14:23:14.092286Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/checknothings with version="0" nonce="8" inc=false
2020-06-03T14:23:14.092343Z    info    mcp    Watch(): created watch 12 for istio/config/v1alpha2/legacy/checknothings from group "default", version "0"
2020-06-03T14:23:14.092390Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/policy/v1beta1/rules with version="10" nonce="9" inc=false
2020-06-03T14:23:14.092423Z    info    mcp    Watch(): created watch 13 for istio/policy/v1beta1/rules from group "default", version "10"
2020-06-03T14:23:14.092478Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/templates with version="0" nonce="10" inc=false
2020-06-03T14:23:14.092509Z    info    mcp    Watch(): created watch 14 for istio/config/v1alpha2/templates from group "default", version "0"
2020-06-03T14:23:14.092567Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/listentries with version="0" nonce="11" inc=false
2020-06-03T14:23:14.092600Z    info    mcp    Watch(): created watch 15 for istio/config/v1alpha2/legacy/listentries from group "default", version "0"
2020-06-03T14:23:14.092647Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/policy/v1beta1/attributemanifests with version="4" nonce="12" inc=false
2020-06-03T14:23:14.092841Z    info    mcp    Watch(): created watch 16 for istio/policy/v1beta1/attributemanifests from group "default", version "4"
2020-06-03T14:23:14.092866Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/quotas with version="0" nonce="13" inc=false
2020-06-03T14:23:14.092896Z    info    mcp    Watch(): created watch 17 for istio/config/v1alpha2/legacy/quotas from group "default", version "0"
2020-06-03T14:23:14.093018Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/policy/v1beta1/instances with version="21" nonce="14" inc=false
2020-06-03T14:23:14.093055Z    info    mcp    Watch(): created watch 18 for istio/policy/v1beta1/instances from group "default", version "21"
2020-06-03T14:23:14.093088Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/edges with version="0" nonce="15" inc=false
2020-06-03T14:23:14.093119Z    info    mcp    Watch(): created watch 19 for istio/config/v1alpha2/legacy/edges from group "default", version "0"
2020-06-03T14:23:14.093544Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/apikeys with version="0" nonce="16" inc=false
2020-06-03T14:23:14.093583Z    info    mcp    Watch(): created watch 20 for istio/config/v1alpha2/legacy/apikeys from group "default", version "0"
2020-06-03T14:23:14.093616Z    info    mcp    MCP: connection {addr=10.132.119.75:49274 id=1} ACK collection=istio/config/v1alpha2/legacy/tracespans with version="0" nonce="17" inc=false
2020-06-03T14:23:14.093649Z    info    mcp    Watch(): created watch 21 for istio/config/v1alpha2/legacy/tracespans from group "default", version "0"
2020-06-03T14:23:14.094603Z    info    mcp    Watch(): created watch 22 for istio/authentication/v1alpha1/policies from group "default", version "0"
2020-06-03T14:23:14.094647Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/networking/v1alpha3/gateways with version="29" nonce="6" inc=false
2020-06-03T14:23:14.094671Z    info    mcp    Watch(): created watch 23 for istio/networking/v1alpha3/gateways from group "default", version "29"
2020-06-03T14:23:14.096363Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/networking/v1alpha3/destinationrules with version="27" nonce="7" inc=false
2020-06-03T14:23:14.096400Z    info    mcp    Watch(): created watch 24 for istio/networking/v1alpha3/destinationrules from group "default", version "27"
2020-06-03T14:23:14.096419Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/networking/v1alpha3/sidecars with version="0" nonce="8" inc=false
2020-06-03T14:23:14.096445Z    info    mcp    Watch(): created watch 25 for istio/networking/v1alpha3/sidecars from group "default", version "0"
2020-06-03T14:23:14.096489Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/rbac/v1alpha1/clusterrbacconfigs with version="0" nonce="9" inc=false
2020-06-03T14:23:14.096515Z    info    mcp    Watch(): created watch 26 for istio/rbac/v1alpha1/clusterrbacconfigs from group "default", version "0"
2020-06-03T14:23:14.096533Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/networking/v1alpha3/virtualservices with version="25" nonce="10" inc=false
2020-06-03T14:23:14.096558Z    info    mcp    Watch(): created watch 27 for istio/networking/v1alpha3/virtualservices from group "default", version "25"
2020-06-03T14:23:14.096601Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/networking/v1alpha3/envoyfilters with version="0" nonce="11" inc=false
2020-06-03T14:23:14.096628Z    info    mcp    Watch(): created watch 28 for istio/networking/v1alpha3/envoyfilters from group "default", version "0"
2020-06-03T14:23:14.096652Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/authentication/v1alpha1/meshpolicies with version="0" nonce="12" inc=false
2020-06-03T14:23:14.096675Z    info    mcp    Watch(): created watch 29 for istio/authentication/v1alpha1/meshpolicies from group "default", version "0"
2020-06-03T14:23:14.096706Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/rbac/v1alpha1/serviceroles with version="0" nonce="13" inc=false
2020-06-03T14:23:14.096730Z    info    mcp    Watch(): created watch 30 for istio/rbac/v1alpha1/serviceroles from group "default", version "0"
2020-06-03T14:23:14.096755Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/rbac/v1alpha1/rbacconfigs with version="0" nonce="14" inc=false
2020-06-03T14:23:14.096778Z    info    mcp    Watch(): created watch 31 for istio/rbac/v1alpha1/rbacconfigs from group "default", version "0"
2020-06-03T14:23:14.096811Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/networking/v1alpha3/serviceentries with version="2" nonce="15" inc=false
2020-06-03T14:23:14.096836Z    info    mcp    Watch(): created watch 32 for istio/networking/v1alpha3/serviceentries from group "default", version "2"
2020-06-03T14:23:14.096860Z    info    mcp    MCP: connection {addr=10.132.178.178:53422 id=2} ACK collection=istio/rbac/v1alpha1/servicerolebindings with version="0" nonce="16" inc=false
2020-06-03T14:23:14.096890Z    info    mcp    Watch(): created watch 33 for istio/rbac/v1alpha1/servicerolebindings from group "default", version "0"
2020-06-03T14:23:14.097013Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3} ACK collection=istio/policy/v1beta1/handlers with version="12" nonce="1" inc=false
2020-06-03T14:23:14.097039Z    info    mcp    Watch(): created watch 34 for istio/policy/v1beta1/handlers from group "default", version "12"
2020-06-03T14:23:14.100499Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3} ACK collection=istio/policy/v1beta1/instances with version="21" nonce="2" inc=false
2020-06-03T14:23:14.100523Z    info    mcp    Watch(): created watch 35 for istio/policy/v1beta1/instances from group "default", version "21"
2020-06-03T14:23:14.100536Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3} ACK collection=istio/policy/v1beta1/rules with version="10" nonce="3" inc=false
2020-06-03T14:23:14.100548Z    info    mcp    Watch(): created watch 36 for istio/policy/v1beta1/rules from group "default", version "10"
2020-06-03T14:23:14.100567Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3} ACK collection=istio/config/v1alpha2/templates with version="0" nonce="4" inc=false
2020-06-03T14:23:14.100578Z    info    mcp    Watch(): created watch 37 for istio/config/v1alpha2/templates from group "default", version "0"
2020-06-03T14:23:14.100588Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3} ACK collection=istio/config/v1alpha2/adapters with version="0" nonce="5" inc=false
2020-06-03T14:23:14.100598Z    info    mcp    Watch(): created watch 38 for istio/config/v1alpha2/adapters from group "default", version "0"
2020-06-03T14:23:14.100612Z    info    mcp    MCP: connection {addr=10.132.66.218:49934 id=3} ACK collection=istio/policy/v1beta1/attributemanifests with version="4" nonce="6" inc=false
2020-06-03T14:23:14.100624Z    info    mcp    Watch(): created watch 39 for istio/policy/v1beta1/attributemanifests from group "default", version "4"
2020-06-03T14:38:24.202406Z    warn    istio.io/istio/galley/pkg/source/kube/dynamic/source.go:126: watch of *unstructured.Unstructured ended with: too old resource version: 82246372 (82315133)
2020-06-03T15:19:14.177111Z    warn    istio.io/istio/galley/pkg/source/kube/dynamic/source.go:126: watch of *unstructured.Unstructured ended with: too old resource version: 82315133 (82335309)
2020-06-03T15:20:41.238554Z    warn    istio.io/istio/galley/pkg/source/kube/dynamic/source.go:126: watch of *unstructured.Unstructured ended with: too old resource version: 82225933 (82336036)
2020-06-03T15:30:43.503093Z    warn    istio.io/istio/galley/pkg/source/kube/dynamic/source.go:126: watch of *unstructured.Unstructured ended with: too old resource version: 82199870 (82341032)
2020-06-03T15:40:32.480569Z    warn    istio.io/istio/galley/pkg/source/kube/dynamic/source.go:126: watch of *unstructured.Unstructured ended with: too old resource version: 82199870 (82345882)

istio-sidecar-injector pod logs:

2020-06-03T12:01:05.439648Z    info    http: TLS handshake error from 10.132.26.79:51404: remote error: tls: bad certificate
2020-06-03T12:02:27.368591Z    info    http: TLS handshake error from 10.132.26.79:52130: remote error: tls: bad certificate
2020-06-03T12:05:11.217933Z    info    http: TLS handshake error from 10.132.26.79:53554: remote error: tls: bad certificate
2020-06-03T12:10:38.907159Z    info    http: TLS handshake error from 10.132.26.79:56434: remote error: tls: bad certificate
2020-06-03T12:21:34.277625Z    info    http: TLS handshake error from 10.132.26.79:33854: remote error: tls: bad certificate

"10.132.26.79" is EKS API server IP.

Istio installed from Helm Chart and works like a charm about 109 days.

[X] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[X] Security
[ ] Test and Release
[X] User Experience
[ ] Developer Infrastructure

Steps to reproduce the bug
Run Istio in EKS until EKS API gets downtime.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

$ istioctl version --remote
client version: 1.3.8
galley version: 1.3.8
ingressgateway version: 1.3.8
ingressgateway version: 1.3.8
ingressgateway version: 1.3.8
pilot version: 1.3.8
policy version: 1.3.8
sidecar-injector version: 1.3.8
telemetry version: 1.3.8

kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-f459c0", GitCommit:"f459c0672169dd35e77af56c24556530a05e9ab1", GitTreeState:"clean", BuildDate:"2020-03-18T04:24:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

helm version
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

How was Istio installed?
Helm

Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS: 1.14

[howardjohn]

Can you examine the cert being used? kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io istio-sidecar-injector -ojsonpath="{.webhooks[0].clientConfig.caBundle}" | base64 -d | openssl x509 -noout -text -in - can do this

[mak-1-sim]

@howardjohn
Thanks for the fast reply, here output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2e:dd:77:13:75:6d:10:77:1e:15:f4:2f:b2:8f:6f:80
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = cluster.local
        Validity
            Not Before: Jan 29 13:05:49 2020 GMT
            Not After : Jan 26 13:05:49 2030 GMT
        Subject: O = cluster.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:ab:86:36:40:e0:4a:f9:1e:4f:51:b4:da:e0:06:
                    a1:8a:bc:df:97:78:3f:72:6d:0a:01:56:f5:46:1f:
                    b1:aa:1c:22:12:01:ea:f9:00:aa:2a:6b:b3:67:30:
                    4e:b9:85:9d:73:b7:0f:31:ae:0b:77:1a:21:8c:a9:
                    b1:fd:89:86:be:a5:1f:27:40:78:7a:ce:15:41:d8:
                    ec:7d:d1:9c:4d:75:79:64:bc:e0:89:30:dc:20:3e:
                    f6:98:63:dc:1a:06:9e:82:f8:31:54:fa:d1:7a:2b:
                    9b:cc:0e:8f:ca:64:bf:7d:ea:6f:44:29:29:13:0c:
                    80:d3:dd:db:b7:51:cd:59:86:71:fa:0a:ab:f3:fc:
                    76:d1:9f:4e:a3:cb:6b:a7:32:a2:c2:99:e7:c2:ea:
                    48:53:1c:3f:fc:bb:89:9b:b2:59:34:2b:a2:82:97:
                    3a:e5:ff:56:6e:ad:21:5c:3c:a3:60:a0:44:bf:c6:
                    d0:b9:8e:4b:0e:df:7f:56:b6:45:28:0d:ca:91:e8:
                    79:ce:db:9d:cc:0f:b4:c8:e9:aa:72:e4:48:5e:4b:
                    88:37:7f:62:61:b3:5b:8d:17:fd:29:64:ce:4d:9a:
                    7e:d5:f4:a5:60:f6:24:45:6b:68:29:60:ac:5a:11:
                    76:95:6c:fc:6e:20:e5:0b:9c:b2:8c:4c:c8:04:5c:
                    1a:87
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         7e:fa:1a:dd:93:a7:eb:a4:30:a1:7b:2b:b2:08:3a:8c:2f:66:
         b7:7c:00:6f:16:71:c9:13:bb:f9:b6:f5:b5:1e:b4:cb:8f:3a:
         ee:7b:b7:4c:6c:3c:a3:6b:da:b2:3d:54:bc:b1:be:4f:0b:9b:
         c6:f2:3c:f3:de:52:4d:84:da:96:e4:74:df:de:0b:2f:c2:21:
         4c:ed:0a:6b:64:9c:3d:50:86:d6:8c:ac:d1:5f:29:51:90:11:
         a2:bd:73:33:f1:a8:85:2a:14:e4:db:7a:c5:d0:07:27:12:be:
         50:d2:2f:4c:fb:0a:32:dd:4c:9f:b4:9d:91:b6:bf:59:58:99:
         6a:45:28:c0:3e:de:17:42:f0:09:c4:73:4a:32:b5:10:2c:f8:
         5d:65:d7:61:b7:79:f3:17:00:95:e2:0b:b3:9a:fc:e5:ee:8b:
         d1:8c:ba:d9:b6:24:c8:28:a3:12:d3:34:30:78:02:32:ef:85:
         a3:51:66:e7:de:ae:5d:73:c4:7d:76:d0:81:9e:77:b4:56:85:
         01:28:f6:02:95:de:7e:8c:72:a1:20:b0:bb:24:1e:43:fb:12:
         18:91:20:bc:c9:cf:08:8c:9a:57:8c:6d:f2:f2:a4:50:e9:96:
         46:a2:d4:d2:36:15:41:a5:c0:34:0f:a0:55:74:a6:b0:be:e8:
         58:ef:ff:97

[howardjohn]

Hm, looks like the cert there is the same as it has been for the last 100 days most likely. I wonder if this means either EKS is somehow not using the cert (seems unlikely) or the injector/galley started serving under a different cert but didn't update the webhook.

Is both injection and validation broken?
Did either restart at any time recently?

[mak-1-sim]

Did either restart at any time recently?

I don't see any non-manual restarts for that pods.

Is both injection and validation broken?

The injection is broken but I not fully sure about validation.

$ kubectl -n istio-system get configmap istio-galley-configuration -o jsonpath='{.data}'
map[validatingwebhookconfiguration.yaml:apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: istio-galley
  labels:
    app: galley
    chart: galley
    heritage: Helm
    release: istio-cp
    istio: galley
webhooks:
  - name: pilot.validation.istio.io
    clientConfig:
      service:
        name: istio-galley
        namespace: istio-system
        path: "/admitpilot"
      caBundle: ""
    rules:
      - operations:
        - CREATE
        - UPDATE
        apiGroups:
        - config.istio.io
        apiVersions:
        - v1alpha2
        resources:
        - httpapispecs
        - httpapispecbindings
        - quotaspecs
        - quotaspecbindings
      - operations:
        - CREATE
        - UPDATE
        apiGroups:
        - rbac.istio.io
        apiVersions:
        - "*"
        resources:
        - "*"
      - operations:
        - CREATE
        - UPDATE
        apiGroups:
        - authentication.istio.io
        apiVersions:
        - "*"
        resources:
        - "*"
      - operations:
        - CREATE
        - UPDATE
        apiGroups:
        - networking.istio.io
        apiVersions:
        - "*"
        resources:
        - destinationrules
        - envoyfilters
        - gateways
        - serviceentries
        - sidecars
        - virtualservices
    failurePolicy: Fail
    sideEffects: None
  - name: mixer.validation.istio.io
    clientConfig:
      service:
        name: istio-galley
        namespace: istio-system
        path: "/admitmixer"
      caBundle: ""
    rules:
      - operations:
        - CREATE
        - UPDATE
        apiGroups:
        - config.istio.io
        apiVersions:
        - v1alpha2
        resources:
        - rules
        - attributemanifests
        - circonuses
        - deniers
        - fluentds
        - kubernetesenvs
        - listcheckers
        - memquotas
        - noops
        - opas
        - prometheuses
        - rbacs
        - solarwindses
        - stackdrivers
        - cloudwatches
        - dogstatsds
        - statsds
        - stdios
        - apikeys
        - authorizations
        - checknothings
        # - kuberneteses
        - listentries
        - logentries
        - metrics
        - quotas
        - reportnothings
        - tracespans
        - adapters
        - handlers
        - instances
        - templates
        - zipkins
    failurePolicy: Fail
    sideEffects: None]

$ kubectl get validatingwebhookconfiguration istio-galley -o yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  creationTimestamp: "2020-02-21T11:57:32Z"
  generation: 1
  labels:
    app: galley
    chart: galley
    heritage: Helm
    istio: galley
    release: istio-cp
  name: istio-galley
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Namespace
    name: istio-system
    uid: e894c68b-4297-11ea-8166-0a428a12de4a
  resourceVersion: "12134547"
  selfLink: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations/istio-galley
  uid: 578d1d6b-54a1-11ea-9703-0259147d2c64
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: <caBundle_here>
    service:
      name: istio-galley
      namespace: istio-system
      path: /admitpilot
  failurePolicy: Fail
  name: pilot.validation.istio.io
  namespaceSelector: {}
  rules:
  - apiGroups:
    - config.istio.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - httpapispecs
    - httpapispecbindings
    - quotaspecs
    - quotaspecbindings
    scope: '*'
  - apiGroups:
    - rbac.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
    scope: '*'
  - apiGroups:
    - authentication.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
    scope: '*'
  - apiGroups:
    - networking.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - destinationrules
    - envoyfilters
    - gateways
    - serviceentries
    - sidecars
    - virtualservices
    scope: '*'
  sideEffects: None
  timeoutSeconds: 30
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: <caBundle_here>
    service:
      name: istio-galley
      namespace: istio-system
      path: /admitmixer
  failurePolicy: Fail
  name: mixer.validation.istio.io
  namespaceSelector: {}
  rules:
  - apiGroups:
    - config.istio.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - rules
    - attributemanifests
    - circonuses
    - deniers
    - fluentds
    - kubernetesenvs
    - listcheckers
    - memquotas
    - noops
    - opas
    - prometheuses
    - rbacs
    - solarwindses
    - stackdrivers
    - cloudwatches
    - dogstatsds
    - statsds
    - stdios
    - apikeys
    - authorizations
    - checknothings
    - listentries
    - logentries
    - metrics
    - quotas
    - reportnothings
    - tracespans
    - adapters
    - handlers
    - instances
    - templates
    - zipkins
    scope: '*'
  sideEffects: None
  timeoutSeconds: 30

[mak-1-sim]

@howardjohn
BTW, mTLS is disabled (from the start) in the mesh.

  mtls:
    enabled: false

[mak-1-sim]

@howardjohn
I found the root cause of the issue!
I updated Istio with:

security:
  enabled: true

And Citadel up and fixed my issue. Looks like it was deployed from the start, but seems someone or me changed enabled: true to enabled: false while updates.

[cloudbow]

I am getting the same error. Where did you add security.enabled=true ?