Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'kubectl logs -f' command dosen't print new logs continuously #81334

Open
xieyanker opened this issue Aug 13, 2019 · 5 comments

Comments

@xieyanker
Copy link

commented Aug 13, 2019

What happened:

Hello, I deploy k8s on mips64. But It only print previous logs, it can't print pod's new logs continuously when I use kubectl logs -f command.
And I view docker's log by docker logs -f, I get normal output.
I also try kubectl logs --follow with kubectl logs --follow=true. Both of them not work.

What you expected to happen:

kubectl logs -f can print pod's log continuously.

How to reproduce it (as minimally and precisely as possible):

I didn't find same issue on amd64 or arm64, it only produce on mips64.
The hyperkube of mips64 is builded by myself, and the kubectl binary is copyed by hyperkube.
Can you give me some iedas?

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/mips64le"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"$Format:%H$", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/mips64le"}
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
NAME="NeoKylin Linux Server"
VERSION="7.0 (loongson)"
ID="neokylin"
ID_LIKE="fedora"
VERSION_ID="7.0"
PRETTY_NAME="NeoKylin Linux Server 7.0 (loongson)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:neokylin:enterprise_linux:7.0:GA:server"
HOME_URL="https://www.cs2c.com.cn/"
BUG_REPORT_URL="https://bugzilla.cs2c.com.cn/"

NEOKYLIN_BUGZILLA_PRODUCT="NeoKylin Linux Server 7.0"
NEOKYLIN_BUGZILLA_PRODUCT_VERSION=7.0
NEOKYLIN_SUPPORT_PRODUCT="NeoKylin Linux Server"
NEOKYLIN_SUPPORT_PRODUCT_VERSION="7.0
  • Kernel (e.g. uname -a):
Linux docker.inspur.com 3.10.0-862.9.1.ns7_4.37.mips64el #1 SMP PREEMPT Tue Feb 26 19:23:39 CST 2019 mips64el mips64el mips64el GNU/Linux
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

@xieyanker xieyanker added the kind/bug label Aug 13, 2019

@xieyanker xieyanker changed the title 'kubectl logs -f' command dosen't print new logs 'kubectl logs -f' command dosen't print new logs continuously Aug 13, 2019

@neolit123

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

/sig cli

@k8s-ci-robot k8s-ci-robot added sig/cli and removed needs-sig labels Aug 13, 2019

@xieyanker

This comment has been minimized.

Copy link
Author

commented Aug 15, 2019

I think the source of the problem is apiserver.

I log the pod with debug mode as follows:

[root@master1]# kubectl logs kube-apiserver-master1 -nkube-system -f --tail=10 -v 6
I0815 19:36:44.682526   16303 loader.go:359] Config loaded from file /root/.kube/config
I0815 19:36:44.941602   16303 round_trippers.go:438] GET https://172.16.3.1:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-master1 200 OK in 226 milliseconds
I0815 19:36:45.214715   16303 round_trippers.go:438] GET https://172.16.3.1:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-master1/log?follow=true&tailLines=10 200 OK in 229 milliseconds

Then I use curl localhost:8080/api/v1/namespaces/kube-system/pods/kube-apiserver-master1/log?follow=true&tailLines=10 to get logs directly, but it cannot log Continuously.

Is there anyone can support some ideas for me?

@xieyanker

This comment has been minimized.

Copy link
Author

commented Aug 15, 2019

I have referenced the issue #54205, but no help for me.

@xieyanker

This comment has been minimized.

Copy link
Author

commented Aug 17, 2019

func waitLogs(ctx context.Context, id string, w *fsnotify.Watcher, runtimeService internalapi.RuntimeService) (bool, error) {
// no need to wait if the pod is not running
if running, err := isContainerRunning(id, runtimeService); !running {
return false, err
}
errRetry := 5
for {
select {
case <-ctx.Done():
return false, fmt.Errorf("context cancelled")
case e := <-w.Events:
switch e.Op {
case fsnotify.Write:
return true, nil
default:
klog.Errorf("Unexpected fsnotify event: %v, retrying...", e)
}
case err := <-w.Errors:
klog.Errorf("Fsnotify watch error: %v, %d error retries remaining", err, errRetry)
if errRetry == 0 {
return false, err
}
errRetry--
case <-time.After(stateCheckPeriod):
if running, err := isContainerRunning(id, runtimeService); !running {
return false, err
}
}
}
}

I have found kubectl logs -f call the waitLogs method, and will into a loop finally.

And I add some debug logs and rebuild the kublet, I found the channel of w.Events will not receive any message on mips64, it will into time.After(stateCheckPeriod) forever, but it can receive message on arm64 with amd64.

@xieyanker

This comment has been minimized.

Copy link
Author

commented Aug 17, 2019

It is fsontify's question.
fsnotify/fsnotify#241
fsnotify/fsnotify#300

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.