Port-forwarding times out

[alexcouper]

Question

• Is there something I'm doing wrong?
• Is port-forward not meant to be used in this way on GKE?

What am I doing?

I have a pod running on GKE that i'd like to connect to 'privately' (ie without exposing it externally). To do this i'm using the port-forward command.

kubectl port-forward -p api-rc-eu9cs 8888:80 --v=10

What's the problem?

The initial request using the forwarded-port seems to be fast, subsequent ones hang and are unreliable. The log has occasional timeouts:
E0102 22:10:52.632087 57705 portforward.go:310] An error occurred forwarding 8888 -> 80: Timed out waiting for data stream

Extra details

I find that kubectl exec occasionally dies during a bash session on a pod, and logs -f too.

Logs

I occasionally get multiple connection messages around the same time

I0102 22:21:53.615357   59005 portforward.go:251] Handling connection for 8888
I0102 22:21:53.619678   59005 portforward.go:251] Handling connection for 8888
I0102 22:21:53.619892   59005 portforward.go:251] Handling connection for 8888

Sometimes the port-forward script dies

E0102 22:20:14.551772   58523 portforward.go:180] Lost connection to pod

Sometimes I get a broken pipe

E0102 22:23:08.032469   59005 portforward.go:310] An error occurred forwarding 8888 -> 80: Error forwarding port 80 to pod ployst-api-rc-5ebd5c5e822a763e9ed8f4dc5e8661b2-eu9cs_default, uid : write |1: broken pipe

But most of the time I get a timeout

E0102 22:22:23.681356   59005 portforward.go:310] An error occurred forwarding 8888 -> 80: Timed out waiting for data stream

[a-robinson]

@cjcullen

[cjcullen]

This sounds like #12179. I have a PR about ready to send out to fix it.

[alexcouper]

@cjcullen is there anything I could do locally to confirm the hypothesis that this is #12179 related?

[cjcullen]

The kubectl exec, lost connection, and broken pipe problems are almost certainly #12179.

I'm not sure about the subsequent request problem. One way to find out would be to launch a GCE cluster and try your steps there. Kubernetes on GCE (as opposed to GKE) doesn't use SSH tunnels for proxying, so if it also fails on GCE, it wouldn't be a tunnel problem.

[samueltauil]

I'm having the same issue here: http://pastebin.com/ENFtqL2G

[cjcullen]

@samueltauil Yours is a slightly different, but related problem. Right now, we default to a 5 minute streaming-conneciton-idle-timeout in kubelet. I sent out #18552 to address that, but I'll probably just end up rolling it in with #19314.

[samueltauil]

@cjcullen any changes that could be defined as a parameter ?

[cjcullen]

Yeah. If you want to pass something higher than 5 minutes (or unlimited) into your kubelets, you can specify the streaming-connection-idle-timeout. E.g.
--streaming-connection-idle-timeout=4h to set it to 4 hours. Or:
--streaming-connection-idle-timeout=0 to make it unlimited.

[cjcullen]

Fixed by #19314