Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optionally use the pod IP to connect to un-intercepted workloads. #3473

Closed
jmalloc opened this issue Jan 2, 2024 · 4 comments
Closed

Optionally use the pod IP to connect to un-intercepted workloads. #3473

jmalloc opened this issue Jan 2, 2024 · 4 comments
Labels
feature New feature or enhancement request

Comments

@jmalloc
Copy link

jmalloc commented Jan 2, 2024

Please describe your use case / problem.

I am trying to work around a 502 Bad Gateway issue that occurs when accessing our application(s) after terminating an intercept. The 502 occurs because the agent is attempting to forward to our container on 127.0.0.1, but our app is bound to the pod's IP address in the 10.x range.

Describe the solution you'd like

An option to proxy via the pod IP, which is already known to the traffic-agent, instead of the loopback interface.

I'm inclined to suggest this should be the default, as the container would have to bind to the pod IP to be accessible under normal circumstances, but understand that would likely break things for others somewhere.

By preference such an option would be configured globally as opposed to per-pod. Perhaps rather than (or in addition to) an explicit option for this behaviour, the agent could auto-detect the best approach.

Describe alternatives you've considered

As far as workarounds:

  1. We have changed some of our applications to bind to 0.0.0.0, such that they accept requests on both the pod IP and the loopback interface. This is mostly OK, but is the first instance we've had to change our applications specifically to work with Telepresence.

  2. Where 1. is problematic or difficult, we manually uninstall the agent after terminating the intercept. The biggest downside to this, other than the the fact that its manual is that it adds an appreciable delay each time the pod is intercepted again.

Versions (please complete the following information)

❯ telepresence version
Client         : v2.18.0
Root Daemon    : v2.18.0
User Daemon    : v2.18.0
Traffic Manager: v2.18.1
❯ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.4-eks-8cb36c9

Additional context

The cause of this 502 took us some time to diagnose as the only error message we were seeing from the traffic agent (after enabling debug logging, IIRC) was delayed connect error: 111. I realise now that this message is from Envoy, but that wasn't obvious to me at the time. I'm not sure if there's anything Telepresence can do to improve this, but thought it was worth mentioning just in case.

@cindymullins-dw cindymullins-dw added the feature New feature or enhancement request label Jan 6, 2024
Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment, or this will be closed in 7 days.

@github-actions github-actions bot added the stale Issue is stale and will be closed label Aug 16, 2024
@jmalloc
Copy link
Author

jmalloc commented Aug 16, 2024

This is still an issue for us.

@thallgren thallgren removed the stale Issue is stale and will be closed label Aug 17, 2024
@thallgren
Copy link
Member

@jmalloc this is now fixed in the OSS version of the traffic-agent, and the fix will be included in the next release. I opted to not make this new behavior configurable, because the implementation should work for all situations that I can think of, and actually fix some that didn't work in the past.

If you'd like to give it a spin, you can try using agent image ghcr.io/telepresenceio/tel2:2.20.0-test.11. One way of installing it is:

telepresence helm install --set agent.image.name=tel2,agent.image.tag=2.20.0-test.11,agent.image.registry=ghcr.io/telepresenceio

Please note that the OSS agent have no support for personal intercepts. The enterprise agent will need some work too in order to support this.

@jmalloc
Copy link
Author

jmalloc commented Aug 27, 2024

This is great, very much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or enhancement request
Projects
None yet
Development

No branches or pull requests

3 participants