Skip to content
This repository has been archived by the owner. It is now read-only.

Admission control webhooks (e.g. sidecar injector) don't work on EKS #271

Closed
fengjian1585 opened this issue Apr 5, 2018 · 74 comments

Comments

@fengjian1585
Copy link

commented Apr 5, 2018

master1 kube-controller-manager: I0405 21:40:29.377013 1538 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"sleep-6bc9d848fc", UID:"322f0a8a-38d5-11e8-aad2-005056846055", APIVersion:"extensions", ResourceVersion:"2633", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

@linsun

This comment has been minimized.

Copy link
Member

commented Apr 5, 2018

could you kindly update the bug with the template?

It seems your sidecar injector pod is not running? can you double check it is running?

cc @ayj @yusuoh

@fengjian1585

This comment has been minimized.

Copy link
Author

commented Apr 5, 2018

sorry,I'm not very useful.

[root@master1 istio-0.7.1]# kubectl -n istio-system get deployment -listio=sidecar-injector
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
istio-sidecar-injector 1 1 1 1 37m

##########

deployment
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
istio-system istio-ca-86f55cc46f-cz2vl 1/1 Running 0 40m 10.233.53.4 node2
istio-system istio-ingress-868d5f978b-ts77q 1/1 Running 0 40m 10.233.63.5 node1
istio-system istio-mixer-65dc5549d6-2zngs 3/3 Running 0 40m 10.233.63.4 node1
istio-system istio-pilot-657cb5ddf7-kw8pv 2/2 Running 0 40m 10.233.53.3 node2
istio-system istio-sidecar-injector-5b8c78fd6-bktlt 1/1 Running 0 37m 10.233.63.6 node1

@ayj

This comment has been minimized.

Copy link

commented Apr 25, 2018

@fengjian1585, is this still an issue?

@ayj ayj removed their assignment Apr 25, 2018

@h4ckroot

This comment has been minimized.

Copy link

commented May 14, 2018

@ayj I have the same issue !

@kirgene

This comment has been minimized.

Copy link

commented May 21, 2018

Also have the same error message (istio-release-0.8-20180519-22-09).

@milosradovanovic

This comment has been minimized.

Copy link

commented May 24, 2018

Any updates on this as I am experiencing the same issue

@sakshigoel12

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2018

@milosradovanovic @kirgene @h4ckroot @fengjian1585 please cloud you upgrade to 0.8 and suggest if you still notice this issue.

@milosradovanovic

This comment has been minimized.

Copy link

commented Jun 5, 2018

@sakshigoel12 I've just checked with 0.8 version and for me still everything is the same, no changes.

@ayj

This comment has been minimized.

Copy link

commented Jun 5, 2018

The following information would be useful to help characterize the nature of this error.

  • k8s version including any provider specific setup instructions (e.g. minikube startup arguments, kops config).

  • New Istio install vs. upgrade?

  • api-server metrics for the webhook

kubectl proxy &
curl -s localhost:8001/metrics | grep sidecar-injector
  • filtered api-server logs
curl -s localhost:8001/logs/kube-apiserver.log | grep sidecar-injector 
  • sidecar-injector pod logs
pod=$(kubectl -n istio-system get pod -listio=sidecar-injector -o jsonpath='{.items[0].metadata.name}')
kubectl -n istio-system logs ${pod}
@saturnism

This comment has been minimized.

Copy link

commented Jun 6, 2018

Same issue here. GKE 1.10.2 Regional Cluster. Istio 0.7.1 w/ Istio Auth.

Pod Log:

2018-06-06T15:41:59.665332Z	info	version root@c5207293dc14-docker.io/istio-0.7.1-62110d4f0373a7613e57b8a4d559ded9cb6a1cc8-Clean
2018-06-06T15:41:59.667144Z	info	New configuration: sha256sum 89f303e89130ed85bd1ec065bde968ac524e134616b9f58552eea01b00505e5d
2018-06-06T15:41:59.667179Z	info	Policy: enabled
2018-06-06T15:41:59.667195Z	info	Template: |
  initContainers:
  - name: istio-init
    image: docker.io/istio/proxy_init:0.7.1
    args:
    - "-p"
    - {{ .MeshConfig.ProxyListenPort }}
    - "-u"
    - 1337
    - -i
    - 10.0.0.0/8
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
    restartPolicy: Always
  containers:
  - name: istio-proxy
    image: docker.io/istio/proxy:0.7.1
    args:
    - proxy
    - sidecar
    - --configPath
    - {{ .ProxyConfig.ConfigPath }}
    - --binaryPath
    - {{ .ProxyConfig.BinaryPath }}
    - --serviceCluster
    {{ if ne "" (index .ObjectMeta.Labels "app") -}}
    - {{ index .ObjectMeta.Labels "app" }}
    {{ else -}}
    - "istio-proxy"
    {{ end -}}
    - --drainDuration
    - {{ formatDuration .ProxyConfig.DrainDuration }}
    - --parentShutdownDuration
    - {{ formatDuration .ProxyConfig.ParentShutdownDuration }}
    - --discoveryAddress
    - {{ .ProxyConfig.DiscoveryAddress }}
    - --discoveryRefreshDelay
    - {{ formatDuration .ProxyConfig.DiscoveryRefreshDelay }}
    - --zipkinAddress
    - {{ .ProxyConfig.ZipkinAddress }}
    - --connectTimeout
    - {{ formatDuration .ProxyConfig.ConnectTimeout }}
    - --statsdUdpAddress
    - {{ .ProxyConfig.StatsdUdpAddress }}
    - --proxyAdminPort
    - {{ .ProxyConfig.ProxyAdminPort }}
    - --controlPlaneAuthPolicy
    - {{ .ProxyConfig.ControlPlaneAuthPolicy }}
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    imagePullPolicy: IfNotPresent
    securityContext:
        privileged: false
        readOnlyRootFilesystem: true
        runAsUser: 1337
    restartPolicy: Always
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      optional: true
      {{ if eq .Spec.ServiceAccountName "" -}}
      secretName: istio.default
      {{ else -}}
      secretName: {{ printf "istio.%s" .Spec.ServiceAccountName }}
      {{ end -}}
@4220182

This comment has been minimized.

Copy link

commented Jun 13, 2018

Same issue here, kubernetes v1.9.6, istio 0.8.0, is New Istio install.

# cat /etc/systemd/system/kube-apiserver.service 
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,ResourceQuota \
  --advertise-address=192.168.5.7 \
  --bind-address=192.168.5.7 \
  --insecure-bind-address=127.0.0.1 \
  --insecure-port=8080 \
  --secure-port=443 \
  --authorization-mode=Node,RBAC \
  --runtime-config=rbac.authorization.k8s.io/v1alpha1 \
  --kubelet-https=true \
  --enable-bootstrap-token-auth \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-cluster-ip-range=10.254.0.0/16 \
  --service-node-port-range=1-60000 \
  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
  --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \
  --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \
  --etcd-servers=https://192.168.5.7:2379,https://192.168.5.8:2379,https://192.168.5.86:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --apiserver-count=1 \
  --endpoint-reconciler-type=lease \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/lib/audit.log \
  --event-ttl=1h \
  --runtime-config=batch/v2alpha1=true \
  --runtime-config=admissionregistration.k8s.io/v1alpha1=true \
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


mac-temp:istio-0.8.0 temp$ kubectl proxy & curl -s localhost:8001/metrics | grep sidecar-injector
Starting to serve on 127.0.0.1:8001
apiserver_admission_webhook_admission_latencies_seconds_bucket{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1",le="25000"} 0
apiserver_admission_webhook_admission_latencies_seconds_bucket{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1",le="62500"} 0
apiserver_admission_webhook_admission_latencies_seconds_bucket{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1",le="156250"} 0
apiserver_admission_webhook_admission_latencies_seconds_bucket{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1",le="390625"} 0
apiserver_admission_webhook_admission_latencies_seconds_bucket{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1",le="976562.5"} 0
apiserver_admission_webhook_admission_latencies_seconds_bucket{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1",le="+Inf"} 4
apiserver_admission_webhook_admission_latencies_seconds_sum{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1"} 1.20006957e+08
apiserver_admission_webhook_admission_latencies_seconds_count{group="",name="sidecar-injector.istio.io",operation="CREATE",rejected="true",resource="pods",subresource="",type="admit",version="v1"} 4
apiserver_request_count{client="sidecar-injector/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",resource="mutatingwebhookconfigurations",scope="cluster",subresource="",verb="GET"} 1
apiserver_request_count{client="sidecar-injector/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",resource="mutatingwebhookconfigurations",scope="cluster",subresource="",verb="PATCH"} 1
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.001"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.002"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.004"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.008"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.016"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.032"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.064"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.128"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.256"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="0.512"} 0
rest_client_request_latency_seconds_bucket{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST",le="+Inf"} 4
rest_client_request_latency_seconds_sum{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST"} 120.00553613299999
rest_client_request_latency_seconds_count{url="https://istio-sidecar-injector.istio-system.svc:443/inject",verb="POST"} 4
rest_client_requests_total{code="<error>",host="istio-sidecar-injector.istio-system.svc:443",method="POST"} 4

mac-temp:istio-0.8.0 temp$ curl -s localhost:8001/logs/kube-apiserver.log | grep sidecar-injector 


mac-temp:istio-0.8.0 temp$ pod=$(kubectl -n istio-system get pod -listio=sidecar-injector -o jsonpath='{.items[0].metadata.name}')
mac-temp:istio-0.8.0 temp$ 
mac-temp:istio-0.8.0 temp$ kubectl -n istio-system logs ${pod}
2018-06-13T11:13:54.600881Z	info	version root@48d5ddfd72da-docker.io/istio-0.8.0-6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84-Clean
2018-06-13T11:13:54.603269Z	info	New configuration: sha256sum 6fb54ce8cab658754b93e2064ea052a1cd0682a6a385ea03d4ebe66707327e2a
2018-06-13T11:13:54.603269Z	info	Policy: enabled
2018-06-13T11:13:54.604163Z	info	Template: |
  initContainers:
  - name: istio-init
    image: docker.io/istio/proxy_init:0.8.0
    args:
    - "-p"
    - [[ .MeshConfig.ProxyListenPort ]]
    - "-u"
    - 1337
    - "-m"
    - [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    - "-i"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges"  ]]"
    [[ else -]]
    - "*"
    [[ end -]]
    - "-x"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges"  ]]"
    [[ else -]]
    - ""
    [[ end -]]
    - "-b"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts"  ]]"
    [[ else -]]
    - [[ range .Spec.Containers -]][[ range .Ports -]][[ .ContainerPort -]], [[ end -]][[ end -]][[ end]]
    - "-d"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts" ]]"
    [[ else -]]
    - ""
    [[ end -]]
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    restartPolicy: Always
  
  containers:
  - name: istio-proxy
    image: [[ if (isset .ObjectMeta.Annotations "sidecar.istio.io/proxyImage") -]]
    "[[ index .ObjectMeta.Annotations "sidecar.istio.io/proxyImage" ]]"
    [[ else -]]
    docker.io/istio/proxyv2:0.8.0
    [[ end -]]
    args:
    - proxy
    - sidecar
    - --configPath
    - [[ .ProxyConfig.ConfigPath ]]
    - --binaryPath
    - [[ .ProxyConfig.BinaryPath ]]
    - --serviceCluster
    [[ if ne "" (index .ObjectMeta.Labels "app") -]]
    - [[ index .ObjectMeta.Labels "app" ]]
    [[ else -]]
    - "istio-proxy"
    [[ end -]]
    - --drainDuration
    - [[ formatDuration .ProxyConfig.DrainDuration ]]
    - --parentShutdownDuration
    - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
    - --discoveryAddress
    - [[ .ProxyConfig.DiscoveryAddress ]]
    - --discoveryRefreshDelay
    - [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]
    - --zipkinAddress
    - [[ .ProxyConfig.ZipkinAddress ]]
    - --connectTimeout
    - [[ formatDuration .ProxyConfig.ConnectTimeout ]]
    - --statsdUdpAddress
    - [[ .ProxyConfig.StatsdUdpAddress ]]
    - --proxyAdminPort
    - [[ .ProxyConfig.ProxyAdminPort ]]
    - --controlPlaneAuthPolicy
    - [[ .ProxyConfig.ControlPlaneAuthPolicy ]]
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: ISTIO_META_INTERCEPTION_MODE
      value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    imagePullPolicy: IfNotPresent
    securityContext:
        privileged: false
        readOnlyRootFilesystem: true
        [[ if eq (or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String) "TPROXY" -]]
        capabilities:
          add:
          - NET_ADMIN
        [[ else -]]
        runAsUser: 1337
        [[ end -]]
    restartPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      optional: true
      [[ if eq .Spec.ServiceAccountName "" -]]
      secretName: istio.default
      [[ else -]]
      secretName: [[ printf "istio.%s" .Spec.ServiceAccountName ]]
      [[ end -]]
2018-06-13T11:13:54.605739Z	warn	Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

@ayj ayj added kind/bug and removed kind/customer issue labels Jun 13, 2018

@ayj ayj self-assigned this Jun 13, 2018

@geeknoid geeknoid removed the kind/bug label Jun 13, 2018

@sprutner

This comment has been minimized.

Copy link

commented Jun 19, 2018

I also have this issue with a new EKS Kubernetes 1.10.3 istio 0.8.0 installation trying to run bookinfo example.

Pod Log:

2018-06-19T17:24:21.338177Z	info	version root@48d5ddfd72da-docker.io/istio-0.8.0-6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84-Clean
2018-06-19T17:24:21.339446Z	info	New configuration: sha256sum 6fb54ce8cab658754b93e2064ea052a1cd0682a6a385ea03d4ebe66707327e2a
2018-06-19T17:24:21.339466Z	info	Policy: enabled
2018-06-19T17:24:21.339491Z	info	Template: |
  initContainers:
  - name: istio-init
    image: docker.io/istio/proxy_init:0.8.0
    args:
    - "-p"
    - [[ .MeshConfig.ProxyListenPort ]]
    - "-u"
    - 1337
    - "-m"
    - [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    - "-i"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges"  ]]"
    [[ else -]]
    - "*"
    [[ end -]]
    - "-x"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges"  ]]"
    [[ else -]]
    - ""
    [[ end -]]
    - "-b"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts"  ]]"
    [[ else -]]
    - [[ range .Spec.Containers -]][[ range .Ports -]][[ .ContainerPort -]], [[ end -]][[ end -]][[ end]]
    - "-d"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts" ]]"
    [[ else -]]
    - ""
    [[ end -]]
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    restartPolicy: Always
  
  containers:
  - name: istio-proxy
    image: [[ if (isset .ObjectMeta.Annotations "sidecar.istio.io/proxyImage") -]]
    "[[ index .ObjectMeta.Annotations "sidecar.istio.io/proxyImage" ]]"
    [[ else -]]
    docker.io/istio/proxyv2:0.8.0
    [[ end -]]
    args:
    - proxy
    - sidecar
    - --configPath
    - [[ .ProxyConfig.ConfigPath ]]
    - --binaryPath
    - [[ .ProxyConfig.BinaryPath ]]
    - --serviceCluster
    [[ if ne "" (index .ObjectMeta.Labels "app") -]]
    - [[ index .ObjectMeta.Labels "app" ]]
    [[ else -]]
    - "istio-proxy"
    [[ end -]]
    - --drainDuration
    - [[ formatDuration .ProxyConfig.DrainDuration ]]
    - --parentShutdownDuration
    - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
    - --discoveryAddress
    - [[ .ProxyConfig.DiscoveryAddress ]]
    - --discoveryRefreshDelay
    - [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]
    - --zipkinAddress
    - [[ .ProxyConfig.ZipkinAddress ]]
    - --connectTimeout
    - [[ formatDuration .ProxyConfig.ConnectTimeout ]]
    - --statsdUdpAddress
    - [[ .ProxyConfig.StatsdUdpAddress ]]
    - --proxyAdminPort
    - [[ .ProxyConfig.ProxyAdminPort ]]
    - --controlPlaneAuthPolicy
    - [[ .ProxyConfig.ControlPlaneAuthPolicy ]]
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: ISTIO_META_INTERCEPTION_MODE
      value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    imagePullPolicy: IfNotPresent
    securityContext:
        privileged: false
        readOnlyRootFilesystem: true
        [[ if eq (or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String) "TPROXY" -]]
        capabilities:
          add:
          - NET_ADMIN
        [[ else -]]
        runAsUser: 1337
        [[ end -]]
    restartPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      optional: true
      [[ if eq .Spec.ServiceAccountName "" -]]
      secretName: istio.default
      [[ else -]]
      secretName: [[ printf "istio.%s" .Spec.ServiceAccountName ]]
      [[ end -]]
2018-06-19T17:24:21.340174Z	warn	Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
@GregoireW

This comment has been minimized.

Copy link

commented Jun 21, 2018

I have the same issue when I was trying EKS.

So K8S 1.10.3, and my setup use the with mutual TLS authentication ( kubectl apply -f install/kubernetes/istio-demo-auth.yaml ) Error like other is :

Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some debug thing I did

  • If I set a namespace without istio-injection=enabled label everything is correct
  • setting the label istio-injection=enabled on a new namespace and deploying ===> ERROR
  • kubectl run ... ==> OK if no injection, KO is injection
  • If I try curl https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s on a pod without injection, result is OK (connection, return someting).

Now my 1st thought:
Is it possible that istio detect that a pod from a namespace should have a proxy, so it enable some kind of egress rule / mutual tls on it, but as the proxy is missing => error ?

@ayj

This comment has been minimized.

Copy link

commented Jun 23, 2018

If I try curl https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s on a pod without injection, result is OK (connection, return something).

Possibly related to istio/istio#6069. You can confirm by checking the caBundle in the istio-sidecar-injector's mutatingwebhookconfiguration (kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml). It should be non-empty.

@sdake

This comment has been minimized.

Copy link
Member

commented Jun 24, 2018

@4220182 This is likely not your problem, however, your kube-apiserver configuration is incorrect. Your runtime-config options are wrong. How did you deploy your Kubernetes system?

@sdake

This comment has been minimized.

Copy link
Member

commented Jun 24, 2018

It seems like a whole slew of people are experiencing this problem on EKS. Has anyone experienced this problem with 0.8.0 of Kubernetes on a different platform?

Cheers
-steve

@sdake

This comment has been minimized.

Copy link
Member

commented Jun 24, 2018

@ayj I can confirm some of @GregoireW 's report.

Reproducer:

  1. deploy EKS
  2. kubectl label namespace default istio-injection=enabled
  3. ~/istio-0.8.0/samples/bookinfo/kube$ kubectl create -f bookinfo.yaml

The bookinfo sample pods do not start and are not visible via kubectl get pods --all-namespaces. I waited for 5-10 minutes for the pods to display.

Unfortunately the control plane for EKS does not appear completely visible e.g.:

sdake@falkor-07:~/istio-0.8.0/samples/bookinfo/kube$ kubectl get pods --all-namespaces
NAMESPACE      NAME                                       READY     STATUS      RESTARTS   AGE
istio-system   grafana-6f6dff9986-qgvwc                   1/1       Running     0          11m
istio-system   istio-citadel-7bdc7775c7-ndjqx             1/1       Running     0          11m
istio-system   istio-cleanup-old-ca-qwkdw                 0/1       Completed   0          11m
istio-system   istio-egressgateway-78dd788b6d-hrk4g       1/1       Running     0          11m
istio-system   istio-ingressgateway-7dd84b68d6-6qq2n      1/1       Running     0          11m
istio-system   istio-mixer-post-install-k6zft             0/1       Completed   0          11m
istio-system   istio-pilot-d5bbc5c59-g5vqb                2/2       Running     0          11m
istio-system   istio-policy-64595c6fff-mqj59              2/2       Running     0          11m
istio-system   istio-sidecar-injector-645c89bc64-nm42l    1/1       Running     0          11m
istio-system   istio-statsd-prom-bridge-949999c4c-xtnqp   1/1       Running     0          11m
istio-system   istio-telemetry-cfb674b6c-n6679            2/2       Running     0          11m
istio-system   istio-tracing-754cdfd695-nprll             1/1       Running     0          11m
istio-system   prometheus-86cb6dd77c-lwzcf                1/1       Running     0          11m
istio-system   servicegraph-5849b7d696-6ddx8              1/1       Running     0          11m
kube-system    aws-node-7wq8j                             1/1       Running     1          15m
kube-system    aws-node-h44q8                             1/1       Running     1          15m
kube-system    aws-node-jl52c                             1/1       Running     0          15m
kube-system    aws-node-tzhrh                             1/1       Running     1          15m
kube-system    kube-dns-7cc87d595-5h5v4                   3/3       Running     0          34m
kube-system    kube-proxy-56nt9                           1/1       Running     0          15m
kube-system    kube-proxy-65p74                           1/1       Running     0          15m
kube-system    kube-proxy-mzbqr                           1/1       Running     0          15m
kube-system    kube-proxy-qr6md                           1/1       Running     0          15m

I'm going to see if I can get some introspection on the control plane next, however, this bug may mostly effect EKS and may be different from bugs others have reported.

It took me awhile to deploy EKS - its not super intuitive and the first 4 or 5 deploys failed completely, so its possible I have something wrong with the environment, although I can confirm the rest of Istio atleast starts up.

Also, this was tested (#271 (comment)) and returned a blob of auth and other metadata (so I think the bundle is still available).

Regards
-steve

@alexmatsak

This comment has been minimized.

Copy link

commented Jun 25, 2018

Tried GKE 1.10.4-gke.2 and istio 0.8.0 (deploy istio manually, not via helm), all works fine with changes from istio/istio#6388

Got client timeout error with EKS 1.10 and istio 0.8.0 (manually and via helm).
Looks like EKS issue

@garysu

This comment has been minimized.

Copy link

commented Aug 23, 2018

@robinjoseph08

This comment has been minimized.

Copy link

commented Sep 25, 2018

It looks like very recently, EKS got support for webhooks (I believe it was added in version eks.2 of the platform, though I can't find any docs for it):

$ kubectl api-versions | grep admissionregistration
admissionregistration.k8s.io/v1beta1

But I tried setting up Istio again (through Helm), and it still failed:

    message: 'Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io":
      Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure

Not sure if there was something else besides the webhooks that was causing it to fail.

@eduardobaitello

This comment has been minimized.

Copy link

commented Sep 25, 2018

It looks like Amazon silently implemented Validation and Mutation webhooks on EKS....

Reading about admission webhooks, I found this post that uses an EKS cluster to explain and test the k8s resources.
So, I tried to use the Istio Sidecar Automatic Injection again, and it's working now...

I installed Istio (v1.0.2) with helm, and I'm deploying my application also using helm.

Helm and k8s version (kubectl config is pointing to my EKS cluster):

$ helm version
Client: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-28T20:13:43Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Helm commands used to install Istio v1.0.2 on EKS:

$ git clone https://github.com/istio/istio.git && cd istio
$ git checkout 1.0.2
$ helm install --wait --name istio --namespace istio-system install/kubernetes/helm/istio --set global.configValidation=true --set sidecarInjectorWebhook.enabled=true

Note: The 1.0.2 tag is mismatching the chart/app version.

After that I just labeled my namespace, and all deployments using helm are being injected with the Istio Sidecars:
$ kubectl label namespace MYNAMESPACE istio-injection=enabled

@sdake

This comment has been minimized.

Copy link
Member

commented Sep 25, 2018

@eduardobaitello we are aware of the helm chart version problem and that will be fixed in 1.0.3. The version field is now automated (iiuc) so that shouldn't happen again.

@ayj

This comment has been minimized.

Copy link

commented Sep 26, 2018

@eduardobaitello, thanks for validating Istio's use of k8s webhooks on EKS. I see that the k8s server version is 1.10.3. Were there any other EKS specific setup instructions or EKS specific version necessary to get things working?

@garysu

This comment has been minimized.

Copy link

commented Sep 26, 2018

@ddbenson

This comment has been minimized.

Copy link

commented Oct 3, 2018

@eduardobaitello You didn't need to modify the ingress/egress gate type (to NodePort) or modify any of the other chart(s) values? The chart just times out for me using your helm install command. What region are you in? Has anyone else been able to confirm this?

@eduardobaitello

This comment has been minimized.

Copy link

commented Oct 3, 2018

@ayj

No EKS specific setup instructions were used. The installation was pretty straightforward, using the templates provided by the Getting Started with Amazon EKS documentation to create the cluster and launch the worker nodes. The EKS Platform Version that I am using is eks.2.

@ddbenson

The EKS cluster region is us-east-1. I didn't modify any configs from Istio Charts.
The istio-ingressgateway k8s service is using the LoadBalancer type, and even automatically created and configured a respective Amazon Load Balancer for me.
Maybe you are missing some of the IAM roles configurations, lacking permissions to set up the Load Balancer, and consequently timing out the helm --wait. Have a try with the templates of Getting Started to create the cluster/nodes.

@dshmatov

This comment has been minimized.

Copy link

commented Oct 9, 2018

@ddbenson I can confirm the same issue as you, helm installation times out. I've got success in installing not using options --set global.configValidation=true --set sidecarInjectorWebhook.enabled=true. But got the negative result while creating pods:

message: 'Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io":
      Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'

Region eu-west-1

@vtrduque

This comment has been minimized.

Copy link

commented Oct 9, 2018

Same of @ddbenson and @dshmatov here. Can't replicate this

Btw, is there a way to use istioctl kube-inject with helm install ?

@dshmatov

This comment has been minimized.

Copy link

commented Oct 9, 2018

Same of @ddbenson and @dshmatov here. Can't replicate this

Btw, is there a way to use istioctl kube-inject with helm install ?

@vtrduque I see the way of injection by using combination helm template and istioctl kube-inject

@vtrduque

This comment has been minimized.

Copy link

commented Oct 9, 2018

@dshmatov Oh, I saw that as well. But I thought that doing something like istioctl kubect-inject -f <(helm template...) | kubectl apply -f - is a really bad smell, besides you lost the utility of versioning of helm 😢 .

Well, I'll follow this thread and wait for the AWS EKS dev team good will haha.

@dshmatov

This comment has been minimized.

Copy link

commented Oct 9, 2018

@vtrduque Yep, It's ugly messy shit :( Still waiting for some reasonable value in EKS admission hooks...

@sdake

This comment has been minimized.

Copy link
Member

commented Oct 9, 2018

I just want to point out EKS as of October is not yet stable with Istio. From my understanding the Amazon EKS team is aware of the problem and working very hard to fix the EKS issues. The good news is once the problems are resolved, other platforms beyond Istio will operate well on EKS. Sadly this isn't very good for folks that need Istio support today on EKS.

Cheers
-steve

@eswarbala

This comment has been minimized.

Copy link

commented Oct 11, 2018

@sdake - Can you expand on the stability issues using istio on EKS?

@sdake

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Hi @eswarbala - Perhaps I spoke incorrectly. My (dated from July) understanding is Amazon is in progress of qualifying Istio on EKS from Shannon McFarland. I have not had an update since I tested in July, so its entirely possible I am incorrect about current status.

I have not run Istio for a few months on EKS. I initially ran Istio 1.0 in July on EKS as a slew of Istio upstream customers were struggling to get EKS running with Istio. This led to feedback to AMZN engineering around a problem set related to missing functionality of the webhooks. If Amazon's customers are satisfied with Istio's functionality on EKS, that works for me and I will remove the warnings from the release notes in the documentation and add a new release note indicating Istio is now functional with EKS.

Can you clarify if AMZN is satisfied with the quality of the current implementation of Istio on EKS?

Cheers
-steve

If you can't discuss here, my email is stdake@cisco.com.

@erks

This comment has been minimized.

@ajohnstone

This comment has been minimized.

Copy link

commented Oct 15, 2018

Tried testing this out with EKS support for dynamic admission controllers, but currently getting the following error, need to look into why...

Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io":
      Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)

From another container...

nc -vz istio-sidecar-injector.istio-system.svc 443
istio-sidecar-injector.istio-system.svc (172.20.241.132:443) open

The error from the sidecar-injection is
failed to convert to proto. unknown field "sdsUdsPath" in v1alpha1.MeshConfig

2018-10-15T08:14:34.564325Z	warn	Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018-10-15T08:22:24.690417Z	error	update error: failed to convert to proto. unknown field "sdsUdsPath" in v1alpha1.MeshConfig
2018-10-15T09:10:07.042974Z	info	http: TLS handshake error from 10.0.3.168:33950: remote error: tls: unknown certificate authority
2018-10-15T09:10:15.731268Z	info	http: TLS handshake error from 10.0.3.168:34000: tls: first record does not look like a TLS handshake
2018-10-15T09:10:41.281410Z	info	http: TLS handshake error from 10.0.3.168:36741: EOF
2018-10-15T09:10:44.381417Z	info	http: TLS handshake error from 10.0.3.168:41115: EOF

Resolved above, which was due to NO_PROXY configuration missing .svc etc.

And now getting

2018-10-15T12:00:26.250122Z	error	Patch webhook failed: the server was unable to return a response in the time allotted, but may still be processing the request (get mutatingwebhookconfigurations.admissionregistration.k8s.io istio-sidecar-injector)
2018-10-15T12:04:49.899607Z	error	Patch webhook failed: the server was unable to return a response in the time allotted, but may still be processing the request (get mutatingwebhookconfigurations.admissionregistration.k8s.io istio-sidecar-injector)
2018-10-15T12:06:23.688602Z	error	Patch webhook failed: the server was unable to return a response in the time allotted, but may still be processing the request (get mutatingwebhookconfigurations.admissionregistration.k8s.io istio-sidecar-injector)
2018-10-15T12:07:51.845052Z	error	Patch webhook failed: the server was unable to return a response in the time allotted, but may still be processing the request (get mutatingwebhookconfigurations.admissionregistration.k8s.io istio-sidecar-injector)
2018-10-15T12:07:54.959876Z	error	Patch webhook failed: Get https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/istio-sidecar-injector: dial tcp 10.96.0.1:443: connect: connection refused
2018-10-15T12:07:55.002993Z	error	Patch webhook failed: Get https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/istio-sidecar-injector: dial tcp 10.96.0.1:443: connect: connection refused
@ddbenson

This comment has been minimized.

Copy link

commented Oct 16, 2018

We got it working. Key was opening up the security groups on our worker nodes.

@ajohnstone

This comment has been minimized.

Copy link

commented Oct 16, 2018

Can confirm working now too

@ajohnstone

This comment has been minimized.

Copy link

commented Oct 16, 2018

Enable port 443 on worker nodes from the master.

Additionally the istio config needed to change to remove sdsUdsPath as came up with a protobuf error.

@dshmatov

This comment has been minimized.

Copy link

commented Oct 18, 2018

Key is in security groups on worker nodes. Can confirm it starts working.

@moshe0076

This comment has been minimized.

Copy link

commented Oct 26, 2018

Can confirm that port 443 must be open from the control plane to the worker nodes.

@eduardobaitello

This comment has been minimized.

Copy link

commented Oct 26, 2018

The security group considerations for EKS is covered here.
There is an AWS CloudFormation template to easily set all these security configs.

@luigi-riefolo

This comment has been minimized.

Copy link

commented Nov 6, 2018

Can we close this issue?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.