Skip to content

kubectl port-forward broken pipe #74551

Open

Description

@dwbrown2

What happened:

We have a pod running in our k8s cluster that I connect to via kubectl port-forward. I am able to connect to this pod (using the following command) but then start getting broken pipe error messages after maintaining a connection for what is typically 30-60s.

kubectl port-forward --namespace monitoring deployment/cost-monitor 9090

There seems to be a correlation between error rate and the amount of data being transferred. I see the following error message initially:

E0225 15:20:06.212139 26392 portforward.go:363] error copying from remote stream to local connection: readfrom tcp6 [::1]:9090->[::1]:57794: write tcp6 [::1]:9090->[::1]:57794: write: broken pipe

These errors are oftentimes followed by timeout messages but necessarily immediately:

E0225 15:22:30.454203 26392 portforward.go:353] error creating forwarding stream for port 9090 -> 9090: Timeout occured

What you expected to happen:
No error messages or major degradation in transfer rate.

How to reproduce it (as minimally and precisely as possible):
Connect via port-forward and transfer ~5 mb over several minutes.

What else?:
We have multiple HTTP requests being made at any given time on this port-forward connection.

Environment:
Experiencing on both AWS kops and GKE

// kops
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

// GKE
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-gke.1", GitCommit:"5c4fddf874319c9825581cc9ab1d0f0cf51e1dc9", GitTreeState:"clean", BuildDate:"2018-11-30T16:18:58Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    on Feb 25, 2019
  2. added
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    on Feb 25, 2019
  3. dwbrown2 commented on Feb 26, 2019

    @dwbrown2
    Author

    /sig network

  4. added
    sig/networkCategorizes an issue or PR as relevant to SIG Network.
    and removed
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    on Feb 26, 2019
  5. AjayTripathy commented on Mar 7, 2019

    @AjayTripathy

    Hi, I’m hitting this on my cluster too.

    Here's what we're currently running:
    Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
    Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6", GitCommit:"b1d75deca493a24a2f87eb1efde1a569e52fc8d9", GitTreeState:"clean", BuildDate:"2018-12-16T04:30:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

    Any additional repro info that I can provide?

  6. added
    sig/cliCategorizes an issue or PR as relevant to SIG CLI.
    triage/unresolvedIndicates an issue that can not or will not be resolved.
    on Mar 7, 2019
  7. added
    sig/api-machineryCategorizes an issue or PR as relevant to SIG API Machinery.
    and removed
    sig/networkCategorizes an issue or PR as relevant to SIG Network.
    on Mar 21, 2019
  8. ysimonson commented on Mar 25, 2019

    @ysimonson

    I'm seeing this issue as well. I think kubelet is not properly closing the error channel associated with port forwarded connections, because I'm seeing the port forwarder get stuck here.

    You don't even need to send a lot of data, just break the reading end of the connection from the client-side while the port forwarder is trying to write. Assuming you're port forwarding an HTTP server available on port 30600 locally, this will reliably reproduce the issue on k8s for docker for mac (server version v1.10.11):

    #!/usr/bin/env python3
    
    import datetime
    import http.client
    
    for _ in range(10):
        conn = http.client.HTTPConnection("localhost:30600")
        conn.request("GET", "/")
        conn.getresponse()
        conn.close()
        print(datetime.datetime.now())
    
    

    I can reliably reproduce the issue on a newer version of k8s + ubuntu + minikube as well, but it is more resilient, and the above script won't trigger the problem. I'm not sure yet how to make a minimally reproducible test for that target yet.

  9. sandeepsharip commented on Apr 18, 2019

    @sandeepsharip

    What's the fix for this

  10. 0xmichalis commented on May 9, 2019

    @0xmichalis
    Contributor

    I have created a simple client+server reproducer at https://github.com/kargakis/k8s-74551-reproducer

    This issue is unfortunate, kubectl port-forward makes it very easy for our less kube-experienced developers to leverage our production deployment for development.

  11. fejta-bot commented on Aug 7, 2019

    @fejta-bot

    Issues go stale after 90d of inactivity.
    Mark the issue as fresh with /remove-lifecycle stale.
    Stale issues rot after an additional 30d of inactivity and eventually close.

    If this issue is safe to close now please do so with /close.

    Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
    /lifecycle stale

  12. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Aug 7, 2019
  13. 110 remaining items

  14. npuichigo commented on Feb 29, 2024

    @npuichigo

    any update here?

  15. sxllwx commented on Mar 4, 2024

    @sxllwx
    Member

    any update here?

    xref: #107203

  16. thesuperzapper commented on Mar 4, 2024

    @thesuperzapper

    @sxllwx just to clarify, it looks like you might be working on a fix (based on your commits), is that correct?

  17. sxllwx commented on Mar 5, 2024

    @sxllwx
    Member

    it looks like you might be working on a fix (based on your commits), is that correct?

    Yes, this problem is caused by the streaming api of the container's runtime (for example: containerd, cri-o). The implementation of the runtime API relies on (in the form of fork or vendor) https://github.com/kubernetes/kubernetes/blob/dc3f5ec6ccb9855dfa99f4c1078625df5fdfab6a/staging/src/k8s.io/kubelet/pkg/cri/streaming/portforward/ httpstream.go

    So we fixed it and added relevant e2e tests . #117493

    According to Mr. aojea's comments, this problem is expected to be fixed in the future. Thanks to @aojea and @brianpursley for their work. 😄

  18. calmacx commented on Jan 6, 2025

    @calmacx

    So this is still a problem on the latest release?
    I'm getting it:

    error copying from remote stream to local connection: readfrom tcp6 [::1]:3000->[::1]:61249: write tcp6 [::1]:3000->[::1]:61249: write: broken pipe
    

    Im on version 1.31.4

  19. montanaflynn commented on Jan 31, 2025

    @montanaflynn

    Nearly 6 years after my original comment in this issue I still run into this a lot.

    I think I've found an okay solution, at least for my needs. Forwarding the port to the pod directly instead of going through the service has been the solution to my connection issues

    # lots of broken pipes and connection issues going through the service
    kubectl port-forward svc/redis 6379:6379 
    
    # connecting to the pod directly, no issues
    kubectl port-forward redis-0 6379:6379 
    
  20. Betulaphobe commented on Mar 25, 2025

    @Betulaphobe

    I'm still getting this issue on client v1.32.3, regardless of whether I'm port-forwarding a pod or a service.

  21. rafagsiqueira commented on May 1, 2025

    @rafagsiqueira

    I am guessing the PR was never merged?

  22. slachiewicz commented on Oct 21, 2026

    @slachiewicz

    The hang after a client disconnect (every new connection failing with error creating error stream … Timeout occurred) is fixed in client-go by #128681, released in kubectl v1.32.0. When the local client goes away, kubectl now resets the data stream instead of leaving it unread, which was stalling every other stream on the shared connection.

    Reproduced on kind v1.37.0 by aborting a 32 MiB download 23 times (3 in sequence, 20 concurrently), through both pod/ and svc/: kubectl v1.31.4 stops forwarding entirely (0/10 later requests succeed); v1.32.0, v1.32.3 and v1.37.1 keep serving (10/10). The fixed versions still log error copying from remote stream to local connection: … broken pipe for each aborted connection; that line is expected and does not mean forwarding broke.

    @Betulaphobe v1.32.3 passes the same test, so the report above may be the log line rather than a hang. If forwarding did stop, the server and container runtime versions would help narrow it down.

    Unless someone reproduces the hang with kubectl v1.32.0 or later, this can be closed as fixed by #128681.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.sig/api-machineryCategorizes an issue or PR as relevant to SIG API Machinery.sig/cliCategorizes an issue or PR as relevant to SIG CLI.triage/unresolvedIndicates an issue that can not or will not be resolved.

Type

No type

Projects

  • Status
    Needs Triage

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions