Kubernetes: requesting flag for "kubectl logs" to avoid 5-minute timeout if no stdout/stderr

[DreadPirateShawn]

/kind feature

What happened:

When running kubectl logs --follow on a pod, after 5 minutes of no stdout/stderr, we received:

$ kubectl --kubeconfig=config --namespace=foo logs --follow foo-oneoff-w8npn --container bar

######################################
#                                    #
#  /system_tests/test_derp.py (1/4)  #
#                                    #
######################################

RESULTS: [/system_tests/test_derp/TestDerp] Ran 4 tests in 130.044 s, ALL TESTS PASSED


######################################
#                                    #
#  /system_tests/test_fuzz.py (2/4)  #
#                                    #
######################################

error: unexpected EOF

What you expected to happen:

The in-progress test was still running, it simply takes about 2 hours -- so after 113 minutes, more stdout was indeed written.

Expected --follow to behave like similar follow flags in (for instance) journalctl or tail -- that is, follow until I ctrl-c to stop.

How to reproduce it (as minimally and precisely as possible):

Run a pod that doesn't output stdout/stderr for over 5 minutes, and attempt to kubectl logs --follow it.

Our particular use-case is, we're using Kubernetes job to run system tests, driven by a Jenkins job. We want to follow the logs so we can see the Jenkins job data in real time, not to mention easily tracking the pod lifetime without needing to poll / check status / sleep / poll / check status / sleep / etc.

Looking at https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs, there's no relevant flags which affect this behavior.

Anything else we need to know?:

Nope.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.8", GitCommit:"d74e09bb4e4e7026f45becbed8310665ddcb8514", GitTreeState:"clean", BuildDate:"2017-08-03T18:12:08Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.8", GitCommit:"d74e09bb4e4e7026f45becbed8310665ddcb8514", GitTreeState:"clean", BuildDate:"2017-08-03T18:01:01Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Bare metal
• OS (e.g. from /etc/os-release): 16.04 LTS (Xenial Xerus)
• Kernel (e.g. uname -a): 4.4.0-89-generic #112-Ubuntu
• Install tools: N/A
• Others: N/A

[k8s-ci-robot]

@DreadPirateShawn: There are no sig labels on this issue. Please add a sig label.

A sig label can be added by either:

1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
   e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

2. specifying the label manually: /sig <group-name>
   e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list.
The <group-suffix> in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

I don't think there a timeout built into kubectl or the apiserver. are you going through a load balancer?

[DreadPirateShawn]

@liggitt Bah! Excellent call, thanks for the insight.

For posterity, the problem here is that we're using haproxy to serve a VIP to balance HA Kubernetes masters.

Specifically we're using HA-Proxy v1.4.21, and we have this in our haproxy cfg:

defaults
  timeout client  600000
  timeout server  600000

Increasing both of those values was required to fix the timeout issue described here.

Closing this ticket.

[dean-luka]

Another option would be to add keep alive packets to kubectl so that connection doesn't die.
Sometimes we don't have control of haproxy (other department), sometimes people in charge for haproxy would refuse to raise timeout, and sometimes it's not even good to raise timeout so much cause it will keep hanging connections longer.
Is it possible to send keep alive packets from kubectl and maybe even configure it inside config file so that we don't have to put it every time on command line - like ssh has keep alive option - ServerAliveInterval?

[cjbottaro]

sometimes people in charge for haproxy would refuse to raise timeout

Yeah, like AWS... the max idle timeout for an A/N/ELB is 4000 seconds.

I'm running a kops cluster in AWS which shoves an ELB in front of the API, and I am frequently running into this issue.

Use cases:

• grepping kubectl logs where there are long periods of inactivity
• Using kubectl exec/run to run a long report, only to come back the next morning to be disconnected

Can we have this issue reopened? Thanks.

[Sure2020]

Yes,like @DreadPirateShawn said:

[nielsbasjes]

@DreadPirateShawn @cjbottaro @liggitt @Sure2020
I ran into this same undesirable effect in k8s 1.18.8.
Because I think this is something everyone running on a high available cluster will run into I created a feature request for this: #94301.

[omitrowski]

Same problem here :)

[mreinhardt]

while true; do kubectl logs -f -lapp=foo | tail -n+10; done  # tail to minimize restart dupe lines

Hacky practicality.

[DreadPirateShawn]

@mreinhardt Could be a viable hack for basic use-cases, but it can easily be messy for others. For instance, if there's also a long log history to begin with, then that loop spews it every time, and that would break tooling that's monitoring for fresh activity.

Sure, you could add timestamps, parse those, de-dupe... but that's kinda the point of requesting a real fix rather than trying to compensate for the bug downstream. :-)