Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS EKS with Calico - Telepresence interception (webhook) not working #3163

Open
peter-galcik opened this issue May 6, 2023 · 0 comments
Open
Labels
feature New feature or enhancement request

Comments

@peter-galcik
Copy link

peter-galcik commented May 6, 2023

Description:
We struggled to find root cause why telepresence inteception for pods didnt work. After thorough investigation and support from your chat channel we were able to find rout cause. The root cause is essentialy same as nicely explained in the blog post here:
https://medium.com/@denisstortisilva/kubernetes-eks-calico-and-custom-admission-webhooks-a2956b49bd0d

In short, as there is known limitation in Calico for EKS where EKS control plane(kubeserver api) is not in same virtual network with kubernetes nodes, webmutating hooksagentInjectorWebhook.yaml (mutatingwebhookconfiguration.admissionregistration.k8s.io/agent-injector-webhook-ambassador) is not reaching kubernetes ambassador service: - agent-injector because service cant be resolved

As this is limiting factor of Calico, it cant be considered as bug in telepresence, but has functional impact, and perhaps better documentation/call diagrams could be available in certain scenarios to enable better troubleshooting in case of unexpected issues.

Solution proposal:
(disclaimer) I am not expert on kubernetes, so iI might not provide best/ideal solutions. I will describe option we have used successfully.

We exposed agent-injector service on nodePort (it means all nodes route thins nodePort which is visible to EKS to dedicated service) afterwards kubernetes-server-api in EKS was able to reach service, but we had to alter webmutating hook as there is https with certificate included to accommodate this as well + changing spec to call url instead of kubernetes service.

I would like to get possibility at least to customize telepresence helm install --values values.yaml in a way that I am able to expose webhook servcie on nodePort

spec:
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: https
      **nodePort: 32666**

And capability of altering webhook configuration (related to agentInjectorWebhook.yaml):

  • certificate contains user defined hostname (as its different than defaults):
    {{- $altNames := <CUSTOM_HOSTNAMES> ( printf "agent-injector.%s" (include "traffic-manager.namespace" .)) ( printf "agent-injector.%s.svc" (include "traffic-manager.namespace" .)) -}}
  • option to use url instead of k8s service:
    service:
      name: {{ .Values.agentInjector.name }}
      namespace: {{ include "traffic-manager.namespace" . }}
      path: {{ .Values.agentInjector.webhook.servicePath }}
      port: {{ .Values.agentInjector.webhook.port }}

to something as this:

  clientConfig:
    caBundle: <bundle>
    url: "https://i<NODE_DNS_NAME>:32666/traffic-agent"

I believe there might be more robust approaach, and I am showing only option we choose. In general, telepresence helm chart lack capability to have externalized (outside) cluster mutating wehbook, which i am not sure is possible to have (as webhook miht need intra cluster information to mutate kubernetes objects).

Workaround:
Manual injection of traffic-agent as per documentation:
https://www.getambassador.io/docs/telepresence/latest/reference/intercepts/manual-agent

Security question:
With approach defined above we are exposing service directly outside cluster. As we run on AWS, there is different layer of security, but it might be questionable in other deployment scenarios.

All respective changes were done on telepresence version 2.12.0

@cindymullins-dw cindymullins-dw added the feature New feature or enhancement request label May 9, 2023
@github-actions github-actions bot added the stale Issue is stale and will be closed label Aug 16, 2024
@thallgren thallgren removed the stale Issue is stale and will be closed label Aug 25, 2024
@telepresenceio telepresenceio deleted a comment from github-actions bot Aug 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or enhancement request
Projects
None yet
Development

No branches or pull requests

3 participants