Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubectl exec through Teleport Proxy results in "broken pipe" error #41014

Closed
programmerq opened this issue Apr 29, 2024 · 3 comments · Fixed by #42091
Closed

Kubectl exec through Teleport Proxy results in "broken pipe" error #41014

programmerq opened this issue Apr 29, 2024 · 3 comments · Fixed by #42091
Assignees
Labels
bug c-vd Internal Customer Reference kubernetes-access tsh tsh - Teleport's command line tool for logging into nodes running Teleport.

Comments

@programmerq
Copy link
Contributor

Expected behavior:

When using tsh kubectl exec to connect to a pod, the connection should be established, allowing the user to perform interactive commands in the pod's container.

Current behavior:

Upon executing tsh kubectl exec, the connection is established, but it results in an error message:

error: write tcp 127.0.0.1:54234->127.0.0.1:54230: write: broken pipe

Bug details:

  • Teleport version: v15.2.2
  • Recreation steps:
    1. Execute tsh kubectl exec -n <namespace> -it <pod-name> -- bash
    2. Receive error message: "error: write tcp 127.0.0.1:54234->127.0.0.1:54230: write: broken pipe"
  • Debug logs:
    {"caller":"reverseproxy/reverse_proxy.go:223","component":"proxy:proxy:kube","level":"info","message":"Round trip: GET https://kube-teleport-proxy-alpn.teleport.cluster.local/api/v1/namespaces/mynamespace/pods/mypod, code: 200, duration: 44.911025ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport.cluster.local","pid":"7.1","timestamp":"2024-04-17T19:37:07Z"}
    {"caller":"proxy/remotecommand.go:164","component":null,"level":"info","message":"Negotiated protocol v4.channel.k8s.io.","timestamp":"2024-04-17T19:37:07Z"}
    {"timestamp":"2024-04-17T19:37:07Z","level":"info","caller":":0","message":"http: response.WriteHeader on hijacked connection from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).WriteHeader (wrap.go:98)"}
    {"timestamp":"2024-04-17T19:37:07Z","level":"info","caller":":0","message":"http: response.Write on hijacked connection from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).Write (wrap.go:80)"}
    {"caller":"proxy/forwarder.go:702","component":"proxy:proxy:kube","level":"warning","message":"Failed writing kube error response body: http: connection has been hijacked","pid":"7.1","timestamp":"2024-04-17T19:37:07Z"}
    

There is a similar issue described in #33020, but the fix for that is already included in v15.2.2.

@programmerq programmerq added bug tsh tsh - Teleport's command line tool for logging into nodes running Teleport. c-vd Internal Customer Reference labels Apr 29, 2024
@DiogoMCampos
Copy link

We're also affected by this issue.

@reynoldsme
Copy link

reynoldsme commented May 20, 2024

We're also seeing this after an upgrade from 14.2.1 to 15.3.1 on a self hosted deployment using TLS routing.

Same errors on the auth server, but for whatever reason it manifests itself as error: Timeout occurred when running kubectl exec. Attempts to run kubectl port-forward result in E0520 11:01:09.764213 34355 portforward.go:234] lost connection to pod.

Downgrading to 14.3.18 resolves the issue for us.

@AntonAM
Copy link
Contributor

AntonAM commented May 21, 2024

Hi, we are working on a fix for this issue. Versions starting from v15.1 are affected. In the meantime as a temporary workaround you can try to use TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE environment variable with a value legacy, like this:

TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE=legacy tsh kubectl exec -n <namespace> -it <pod-name> -- bash

You can also use it with the tsh proxy kube command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug c-vd Internal Customer Reference kubernetes-access tsh tsh - Teleport's command line tool for logging into nodes running Teleport.
Projects
None yet
5 participants