Cannot exec into a pod when using tsh proxy kube: Failed writing kube error response body: http: connection has been hijacked

[ravicious]

Expected behavior:

After starting a kube proxy and pointing my KUBECONFIG at the generated file, I should be able to exec into a pod.

Current behavior:

After starting a kube proxy and using its KUBECONFIG, kubectl exec fails with Error from server: and in the server logs I see this:

2023-10-05T17:05:16+02:00 INFO [KUBERNETE] Round trip: GET https://127.0.0.1:54626/api/v1/namespaces/default/pods/hello-node-67949d9db-vcqvg, code: 200, duration: 20.649ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport.cluster.local pid:40520.1 reverseproxy/reverse_proxy.go:236
2023-10-05T17:05:16+02:00 INFO [PROXY:PRO] Round trip: GET https://kube-teleport-proxy-alpn.teleport.cluster.local/api/v1/namespaces/default/pods/hello-node-67949d9db-vcqvg, code: 200, duration: 21.526ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport-local.dev pid:40520.1 reverseproxy/reverse_proxy.go:236
2023-10-05T17:05:16+02:00 INFO [AUDIT]     kube.request addr.remote:127.0.0.1:63340 cluster_name:teleport-local code:T3009I ei:0 event:kube.request kubernetes_cluster:minikube kubernetes_groups:[admins viewers system:authenticated] kubernetes_users:[minikube] login:rav namespace:default proto:kube request_path:/api/v1/namespaces/default/pods/hello-node-67949d9db-vcqvg resource_api_group:core/v1 resource_kind:pods resource_name:hello-node-67949d9db-vcqvg resource_namespace:default response_code:200 server_id:3fc216dd-b53d-4ebe-849b-3e2eaad62668 time:2023-10-05T15:05:16.911Z uid:410e360d-6c88-4906-8b07-9ab4194153ae user:rav verb:GET events/emitter.go:274
2023-10-05T17:05:16+02:00 INFO             Negotiated protocol v4.channel.k8s.io. proxy/remotecommand.go:152
2023/10/05 17:05:16 http: response.WriteHeader on hijacked connection from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).WriteHeader (wrap.go:98)
2023/10/05 17:05:16 http: response.Write on hijacked connection from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).Write (wrap.go:80)
2023-10-05T17:05:16+02:00 WARN [PROXY:PRO] Failed writing kube error response body: http: connection has been hijacked pid:40520.1 proxy/forwarder.go:718
2023-10-05T17:06:14+02:00 WARN [FILE]      Skipping upload 6b5a8c52-c923-4bef-b962-1d41616e469d, missing subdirectory. filesessions/filestream.go:267
2023-10-05T17:06:19+02:00 INFO [AUDIT]     user.login addr.remote:127.0.0.1:63828 cluster_name:teleport-local code:T1000I ei:0 event:user.login method:local success:true time:2023-10-05T15:06:19.892Z uid:374f4c82-2009-4288-ac4a-db181cae139c user:rav user_agent:Go-http-client/1.1 events/emitter.go:274
2023-10-05T17:06:19+02:00 INFO [CA]        Generating TLS certificate 1.3.9999.1.15=#13046e6f6e65,1.3.9999.2.6=#1308706f737467726573,1.3.9999.2.6=#0c0c6462615f726561646f6e6c79,1.3.9999.2.6=#1303646261,1.3.9999.2.6=#0c012a,1.3.9999.2.5=#0c012a,1.3.9999.1.9=#13093132372e302e302e31,1.3.9999.1.7=#130e74656c65706f72742d6c6f63616c,1.3.9999.1.3=#13086d696e696b756265,1.3.9999.1.2=#130776696577657273,1.3.9999.1.2=#130661646d696e73,1.3.9999.1.1=#13086d696e696b756265,CN=rav,O=access+O=connect-my-computer-rav+O=db+O=editor+O=parallels,POSTALCODE={\"aws_role_arns\":null\,\"db_names\":null\,\"db_users\":null\,\"kubernetes_groups\":null\,\"kubernetes_users\":null\,\"logins\":[\"rav\"\,\"dummy\"\,\"dummy2\"]\,\"windows_logins\":null},STREET=teleport-local,L=rav+L=dummy+L=dummy2+L=root+L=parallels+L=-teleport-internal-join dns_names:[] key_usage:5 not_after:2023-10-05 23:06:19.894531 +0000 UTC tlsca/ca.go:1119
2023-10-05T17:06:19+02:00 INFO [AUDIT]     cert.create cert_type:user cluster_name:teleport-local code:TC000I ei:0 event:cert.create client_ip:127.0.0.1 database_names:[*] database_users:[* dba dba_readonly postgres] expires:2023-10-05T23:06:19.894531Z kubernetes_cluster:minikube kubernetes_groups:[admins viewers] kubernetes_users:[minikube] logins:[rav dummy dummy2 root parallels -teleport-internal-join] prev_identity_expires:0001-01-01T00:00:00Z roles:[access connect-my-computer-rav db editor parallels] route_to_cluster:teleport-local teleport_cluster:teleport-local aws_role_arns:<nil> db_names:<nil> db_users:<nil> kubernetes_groups:<nil> kubernetes_users:<nil> logins:[rav dummy dummy2] windows_logins:<nil> user:rav time:2023-10-05T15:06:19.896Z uid:efdb8ac6-7127-4063-9c76-d444ef970d2a events/emitter.go:274
2023-10-05T17:06:26+02:00 INFO [CA]        Generating TLS certificate 1.3.9999.1.15=#13046e6f6e65,1.3.9999.2.6=#1308706f737467726573,1.3.9999.2.6=#0c0c6462615f726561646f6e6c79,1.3.9999.2.6=#1303646261,1.3.9999.2.6=#0c012a,1.3.9999.2.5=#0c012a,1.3.9999.1.9=#13093132372e302e302e31,1.3.9999.1.7=#130e74656c65706f72742d6c6f63616c,1.3.9999.1.3=#13086d696e696b756265,1.3.9999.1.2=#130776696577657273,1.3.9999.1.2=#130661646d696e73,1.3.9999.1.1=#13086d696e696b756265,CN=rav,OU=usage:kube,O=db+O=access+O=editor+O=parallels+O=connect-my-computer-rav,POSTALCODE={\"aws_role_arns\":null\,\"db_names\":null\,\"db_users\":null\,\"kubernetes_groups\":null\,\"kubernetes_users\":null\,\"logins\":[\"rav\"\,\"dummy\"\,\"dummy2\"]\,\"windows_logins\":null},STREET=teleport-local,L=rav+L=dummy+L=dummy2+L=root+L=parallels+L=-teleport-internal-join dns_names:[] key_usage:5 not_after:2023-10-05 23:06:19.001714 +0000 UTC tlsca/ca.go:1119
2023-10-05T17:06:26+02:00 INFO [AUDIT]     cert.create cert_type:user cluster_name:teleport-local code:TC000I ei:0 event:cert.create client_ip:127.0.0.1 database_names:[*] database_users:[* dba dba_readonly postgres] expires:2023-10-05T23:06:19.001714Z kubernetes_cluster:minikube kubernetes_groups:[admins viewers] kubernetes_users:[minikube] logins:[rav dummy dummy2 root parallels -teleport-internal-join] prev_identity_expires:0001-01-01T00:00:00Z roles:[db access editor parallels connect-my-computer-rav] route_to_cluster:teleport-local teleport_cluster:teleport-local aws_role_arns:<nil> db_names:<nil> db_users:<nil> kubernetes_groups:<nil> kubernetes_users:<nil> logins:[rav dummy dummy2] windows_logins:<nil> usage:[usage:kube] user:rav time:2023-10-05T15:06:26.375Z uid:e99abed9-0da7-470a-a88a-3f31065144a9 events/emitter.go:274
2023-10-05T17:06:38+02:00 INFO [KUBERNETE] Round trip: GET https://127.0.0.1:54626/api/v1/namespaces/default/pods/hello-node-67949d9db-vcqvg, code: 200, duration: 12.867ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport.cluster.local pid:40520.1 reverseproxy/reverse_proxy.go:236
2023-10-05T17:06:38+02:00 INFO [PROXY:PRO] Round trip: GET https://kube-teleport-proxy-alpn.teleport.cluster.local/api/v1/namespaces/default/pods/hello-node-67949d9db-vcqvg, code: 200, duration: 14.549ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport-local.dev pid:40520.1 reverseproxy/reverse_proxy.go:236
2023-10-05T17:06:38+02:00 INFO [AUDIT]     kube.request addr.remote:127.0.0.1:63921 cluster_name:teleport-local code:T3009I ei:0 event:kube.request kubernetes_cluster:minikube kubernetes_groups:[admins viewers system:authenticated] kubernetes_users:[minikube] login:rav namespace:default proto:kube request_path:/api/v1/namespaces/default/pods/hello-node-67949d9db-vcqvg resource_api_group:core/v1 resource_kind:pods resource_name:hello-node-67949d9db-vcqvg resource_namespace:default response_code:200 server_id:3fc216dd-b53d-4ebe-849b-3e2eaad62668 time:2023-10-05T15:06:38.919Z uid:ee44c2ca-bf7f-4141-8f4a-b6f7a75bda3d user:rav verb:GET events/emitter.go:274
2023-10-05T17:06:38+02:00 INFO             Negotiated protocol v4.channel.k8s.io. proxy/remotecommand.go:152
2023/10/05 17:06:38 http: response.WriteHeader on hijacked connection from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).WriteHeader (wrap.go:98)
2023/10/05 17:06:38 http: response.Write on hijacked connection from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).Write (wrap.go:80)
2023-10-05T17:06:38+02:00 WARN [PROXY:PRO] Failed writing kube error response body: http: connection has been hijacked pid:40520.1 proxy/forwarder.go:718

Originally reported on community Slack, I was able to reproduce it.

Bug details:

• Teleport version: master, v14.

[codekoala]

We are seeing this as well. We just upgraded from Teleport 13.3.8 to 14.0.1. We are running on Kubernetes 1.26.4 if that makes any difference.