Repository navigation
Description
What happened:
We have a pod running in our k8s cluster that I connect to via kubectl port-forward. I am able to connect to this pod (using the following command) but then start getting broken pipe error messages after maintaining a connection for what is typically 30-60s.
kubectl port-forward --namespace monitoring deployment/cost-monitor 9090
There seems to be a correlation between error rate and the amount of data being transferred. I see the following error message initially:
E0225 15:20:06.212139 26392 portforward.go:363] error copying from remote stream to local connection: readfrom tcp6 [::1]:9090->[::1]:57794: write tcp6 [::1]:9090->[::1]:57794: write: broken pipe
These errors are oftentimes followed by timeout messages but necessarily immediately:
E0225 15:22:30.454203 26392 portforward.go:353] error creating forwarding stream for port 9090 -> 9090: Timeout occured
What you expected to happen:
No error messages or major degradation in transfer rate.
How to reproduce it (as minimally and precisely as possible):
Connect via port-forward and transfer ~5 mb over several minutes.
What else?:
We have multiple HTTP requests being made at any given time on this port-forward connection.
Environment:
Experiencing on both AWS kops and GKE
// kops
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
// GKE
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-gke.1", GitCommit:"5c4fddf874319c9825581cc9ab1d0f0cf51e1dc9", GitTreeState:"clean", BuildDate:"2018-11-30T16:18:58Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsNeeds Triage
Activity
dwbrown2 commentedon Feb 26, 2019
/sig network
AjayTripathy commentedon Mar 7, 2019
Hi, I’m hitting this on my cluster too.
Here's what we're currently running:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6", GitCommit:"b1d75deca493a24a2f87eb1efde1a569e52fc8d9", GitTreeState:"clean", BuildDate:"2018-12-16T04:30:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Any additional repro info that I can provide?
ysimonson commentedon Mar 25, 2019
I'm seeing this issue as well. I think kubelet is not properly closing the error channel associated with port forwarded connections, because I'm seeing the port forwarder get stuck here.
You don't even need to send a lot of data, just break the reading end of the connection from the client-side while the port forwarder is trying to write. Assuming you're port forwarding an HTTP server available on port 30600 locally, this will reliably reproduce the issue on k8s for docker for mac (server version v1.10.11):
I can reliably reproduce the issue on a newer version of k8s + ubuntu + minikube as well, but it is more resilient, and the above script won't trigger the problem. I'm not sure yet how to make a minimally reproducible test for that target yet.
sandeepsharip commentedon Apr 18, 2019
What's the fix for this
0xmichalis commentedon May 9, 2019
I have created a simple client+server reproducer at https://github.com/kargakis/k8s-74551-reproducer
This issue is unfortunate,
kubectl port-forwardmakes it very easy for our less kube-experienced developers to leverage our production deployment for development.fejta-bot commentedon Aug 7, 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
110 remaining items
npuichigo commentedon Feb 29, 2024
any update here?
sxllwx commentedon Mar 4, 2024
xref: #107203
thesuperzapper commentedon Mar 4, 2024
@sxllwx just to clarify, it looks like you might be working on a fix (based on your commits), is that correct?
sxllwx commentedon Mar 5, 2024
Yes, this problem is caused by the streaming api of the container's runtime (for example: containerd, cri-o). The implementation of the runtime API relies on (in the form of fork or vendor) https://github.com/kubernetes/kubernetes/blob/dc3f5ec6ccb9855dfa99f4c1078625df5fdfab6a/staging/src/k8s.io/kubelet/pkg/cri/streaming/portforward/ httpstream.go
So we fixed it and added relevant e2e tests . #117493
According to Mr. aojea's comments, this problem is expected to be fixed in the future. Thanks to @aojea and @brianpursley for their work. 😄
bugfix(port-forward): Correctly handle known errors
Add e2e test reproducing issue kubernetes#74551
Add e2e test reproducing issue kubernetes#74551
Add e2e test reproducing issue kubernetes#74551
Add e2e test reproducing issue kubernetes#74551
Add e2e test reproducing issue kubernetes#74551
calmacx commentedon Jan 6, 2025
So this is still a problem on the latest release?
I'm getting it:
Im on version
1.31.4montanaflynn commentedon Jan 31, 2025
Nearly 6 years after my original comment in this issue I still run into this a lot.
I think I've found an okay solution, at least for my needs. Forwarding the port to the pod directly instead of going through the service has been the solution to my connection issues
Betulaphobe commentedon Mar 25, 2025
I'm still getting this issue on client v1.32.3, regardless of whether I'm port-forwarding a
podor aservice.rafagsiqueira commentedon May 1, 2025
I am guessing the PR was never merged?
slachiewicz commentedon Oct 21, 2026
The hang after a client disconnect (every new connection failing with
error creating error stream … Timeout occurred) is fixed in client-go by #128681, released in kubectl v1.32.0. When the local client goes away, kubectl now resets the data stream instead of leaving it unread, which was stalling every other stream on the shared connection.Reproduced on kind v1.37.0 by aborting a 32 MiB download 23 times (3 in sequence, 20 concurrently), through both
pod/andsvc/: kubectl v1.31.4 stops forwarding entirely (0/10 later requests succeed); v1.32.0, v1.32.3 and v1.37.1 keep serving (10/10). The fixed versions still logerror copying from remote stream to local connection: … broken pipefor each aborted connection; that line is expected and does not mean forwarding broke.@Betulaphobe v1.32.3 passes the same test, so the report above may be the log line rather than a hang. If forwarding did stop, the server and container runtime versions would help narrow it down.
Unless someone reproduces the hang with kubectl v1.32.0 or later, this can be closed as fixed by #128681.