Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubectl logs do not always work in 1.9.0 #57359

Closed
szaharici opened this issue Dec 18, 2017 · 3 comments

Comments

@szaharici
Copy link

commented Dec 18, 2017

/kind bug
/sig api-machinery

What happened:
kubectl logs is not always able to retrieve the logs of pods
What you expected to happen:
kubectl logs would show the logs of running pods
How to reproduce it (as minimally and precisely as possible):

kubectl get pods -n kube-system
NAME                                    READY     STATUS    RESTARTS   AGE
heapster-dfd674df9-r49f9                1/1       Running   0          5m
kube-dns-5d585466b5-ghxk2               3/3       Running   0          4h
kubernetes-dashboard-7494579689-tbwln   1/1       Running   0          5m
monitoring-grafana-76848b566c-xl8n9     1/1       Running   0          5m
monitoring-influxdb-6c4b84d695-7452w    1/1       Running   0          5m
tiller-deploy-8dc84bcc8-9brjg           1/1       Running   0          3h
kubectl -n kube-system logs heapster-dfd674df9-r49f9
Error from server (NotFound): the server could not find the requested resource ( pods/log heapster-dfd674df9-r49f9)

Anything else we need to know?:
This is a new installation of kubernetes 1.9.0 ( the master and three nodes are running 1.9.0).
Even though it is not possible to see the logs the containers seem to run fine.
Logs can be viewed on the nodes that hold the containers by running docker logs.
Some of the pod logs can be viewed
Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): CentOS Linux release 7.4.1708
  • Kernel (e.g. uname -a): Linux kubernetes-1 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:
@szaharici szaharici changed the title kubectl logs does not always work in 1.9.0 kubectl logs do not always work in 1.9.0 Dec 19, 2017
@szaharici

This comment has been minimized.

Copy link
Author

commented Dec 19, 2017

This was caused by a mis-configuration in the kubelet configuration on two nodes. --hostname-override pointed to a single node name

@szaharici szaharici closed this Dec 19, 2017
@thewalk

This comment has been minimized.

Copy link

commented Jan 25, 2018

are you saying that the kubelet need to be updated?

@szaharici

This comment has been minimized.

Copy link
Author

commented Jan 25, 2018

No, it was just a misconfiguration, all the nodes had a hostname-override pointing to the same node therefore preventing the retrieval of the logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.