Admission control webhooks (e.g. sidecar injector) don't work on EKS #271
Comments
|
sorry，I'm not very useful. [root@master1 istio-0.7.1]# kubectl -n istio-system get deployment -listio=sidecar-injector ########## deployment |
|
@fengjian1585, is this still an issue? |
|
@ayj I have the same issue ! |
|
Also have the same error message (istio-release-0.8-20180519-22-09). |
|
Any updates on this as I am experiencing the same issue |
|
@milosradovanovic @kirgene @h4ckroot @fengjian1585 please cloud you upgrade to 0.8 and suggest if you still notice this issue. |
|
@sakshigoel12 I've just checked with 0.8 version and for me still everything is the same, no changes. |
|
The following information would be useful to help characterize the nature of this error.
kubectl proxy &
curl -s localhost:8001/metrics | grep sidecar-injector
curl -s localhost:8001/logs/kube-apiserver.log | grep sidecar-injector
pod=$(kubectl -n istio-system get pod -listio=sidecar-injector -o jsonpath='{.items[0].metadata.name}')
kubectl -n istio-system logs ${pod} |
|
Same issue here. GKE 1.10.2 Regional Cluster. Istio 0.7.1 w/ Istio Auth. Pod Log: |
|
Same issue here, kubernetes v1.9.6, istio 0.8.0, is New Istio install. |
|
I also have this issue with a new EKS Kubernetes 1.10.3 istio 0.8.0 installation trying to run bookinfo example. Pod Log: |
|
I have the same issue when I was trying EKS. So K8S 1.10.3, and my setup use the with mutual TLS authentication (
Some debug thing I did
Now my 1st thought: |
Possibly related to istio/istio#6069. You can confirm by checking the |
|
@4220182 This is likely not your problem, however, your kube-apiserver configuration is incorrect. Your runtime-config options are wrong. How did you deploy your Kubernetes system? |
|
It seems like a whole slew of people are experiencing this problem on EKS. Has anyone experienced this problem with 0.8.0 of Kubernetes on a different platform? Cheers |
|
@ayj I can confirm some of @GregoireW 's report. Reproducer:
The bookinfo sample pods do not start and are not visible via Unfortunately the control plane for EKS does not appear completely visible e.g.: I'm going to see if I can get some introspection on the control plane next, however, this bug may mostly effect EKS and may be different from bugs others have reported. It took me awhile to deploy EKS - its not super intuitive and the first 4 or 5 deploys failed completely, so its possible I have something wrong with the environment, although I can confirm the rest of Istio atleast starts up. Also, this was tested (#271 (comment)) and returned a blob of auth and other metadata (so I think the bundle is still available). Regards |
|
Tried GKE 1.10.4-gke.2 and istio 0.8.0 (deploy istio manually, not via helm), all works fine with changes from istio/istio#6388 Got client timeout error with EKS 1.10 and istio 0.8.0 (manually and via helm). |
|
It looks like Amazon silently implemented Validation and Mutation webhooks on EKS.... Reading about admission webhooks, I found this post that uses an EKS cluster to explain and test the k8s resources. I installed Istio (v1.0.2) with helm, and I'm deploying my application also using helm. Helm and k8s version (kubectl config is pointing to my EKS cluster): Helm commands used to install Istio v1.0.2 on EKS: Note: The 1.0.2 tag is mismatching the chart/app version. After that I just labeled my namespace, and all deployments using helm are being injected with the Istio Sidecars: |
|
@eduardobaitello we are aware of the helm chart version problem and that will be fixed in 1.0.3. The version field is now automated (iiuc) so that shouldn't happen again. |
|
@eduardobaitello, thanks for validating Istio's use of k8s webhooks on EKS. I see that the k8s server version is 1.10.3. Were there any other EKS specific setup instructions or EKS specific version necessary to get things working? |
|
I would be careful here as those webhooks were not officially included in the EKS v2 release they just did. Getting this working and officially supported is at the top of their priority list and officially support is coming before the end of the year.
From: Jason Young <notifications@github.com>
Sent: Tuesday, September 25, 2018 5:23 PM
To: istio/old_issues_repo <old_issues_repo@noreply.github.com>
Cc: Gary Sumner <garysu@seventh-symbol.net>; Mention <mention@noreply.github.com>
Subject: Re: [istio/old_issues_repo] Admission control webhooks (e.g. sidecar injector) don't work on EKS (#271)
@eduardobaitello<https://github.com/eduardobaitello>, thanks for validating Istio's use of k8s webhooks on EKS. I see that the k8s server version is 1.10.3. Were there any other EKS specific setup instructions or EKS specific version necessary to get things working?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#271 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABzuQ7-kwRFsSVmadOc_fX2RQ6Elqfjdks5uesjLgaJpZM4TIeWP>.
|
|
@eduardobaitello You didn't need to modify the ingress/egress gate type (to NodePort) or modify any of the other chart(s) values? The chart just times out for me using your helm install command. What region are you in? Has anyone else been able to confirm this? |
|
No EKS specific setup instructions were used. The installation was pretty straightforward, using the templates provided by the Getting Started with Amazon EKS documentation to create the cluster and launch the worker nodes. The EKS Platform Version that I am using is The EKS cluster region is |
|
@ddbenson I can confirm the same issue as you, helm installation times out. I've got success in installing not using options Region |
|
@dshmatov Oh, I saw that as well. But I thought that doing something like Well, I'll follow this thread and wait for the AWS EKS dev team good will haha. |
|
@vtrduque Yep, It's ugly messy shit :( Still waiting for some reasonable value in EKS admission hooks... |
|
I just want to point out EKS as of October is not yet stable with Istio. From my understanding the Amazon EKS team is aware of the problem and working very hard to fix the EKS issues. The good news is once the problems are resolved, other platforms beyond Istio will operate well on EKS. Sadly this isn't very good for folks that need Istio support today on EKS. Cheers |
|
@sdake - Can you expand on the stability issues using istio on EKS? |
|
Hi @eswarbala - Perhaps I spoke incorrectly. My (dated from July) understanding is Amazon is in progress of qualifying Istio on EKS from Shannon McFarland. I have not had an update since I tested in July, so its entirely possible I am incorrect about current status. I have not run Istio for a few months on EKS. I initially ran Istio 1.0 in July on EKS as a slew of Istio upstream customers were struggling to get EKS running with Istio. This led to feedback to AMZN engineering around a problem set related to missing functionality of the webhooks. If Amazon's customers are satisfied with Istio's functionality on EKS, that works for me and I will remove the warnings from the release notes in the documentation and add a new release note indicating Istio is now functional with EKS. Can you clarify if AMZN is satisfied with the quality of the current implementation of Istio on EKS? Cheers If you can't discuss here, my email is stdake@cisco.com. |
|
https://aws.amazon.com/about-aws/whats-new/2018/10/amazon-eks-enables-support-for-kubernetes-dynamic-admission-cont/ it should work now (though I haven't tried it). |
|
Tried testing this out with EKS support for dynamic admission controllers, but currently getting the following error, need to look into why... From another container... The error from the sidecar-injection is Resolved above, which was due to NO_PROXY configuration missing .svc etc. And now getting |
|
We got it working. Key was opening up the security groups on our worker nodes. |
|
Can confirm working now too |
|
Enable port 443 on worker nodes from the master. Additionally the istio config needed to change to remove sdsUdsPath as came up with a protobuf error. |
|
Key is in security groups on worker nodes. Can confirm it starts working. |
|
Can confirm that port 443 must be open from the control plane to the worker nodes. |
|
The security group considerations for EKS is covered here. |
|
Can we close this issue? |
and others
master1 kube-controller-manager: I0405 21:40:29.377013 1538 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"sleep-6bc9d848fc", UID:"322f0a8a-38d5-11e8-aad2-005056846055", APIVersion:"extensions", ResourceVersion:"2633", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
The text was updated successfully, but these errors were encountered: