Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

Closed
obiyuta opened this issue Jun 28, 2018 · 32 comments
Closed

Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

obiyuta opened this issue Jun 28, 2018 · 32 comments
Assignees
Labels

Comments

@obiyuta
Copy link

@obiyuta obiyuta commented Jun 28, 2018

/kind bug

What happened:

When getting nodes or pods using the command kubectl get pods kubectl get pods, the following output appears:

No resources found.
Error from server (NotAcceptable): unknown (get nodes)
No resources found.
Error from server (NotAcceptable): unknown (get pods)

What you expected to happen:

When running the command, kubectl get pods/nodes , with kubectl version 1.11.0, pods/nodes appear.

How to reproduce it (as minimally and precisely as possible):

  1. Run kubectl get pods using kubectl 1.11.0

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release):
  • Kernel:
Linux default-52e963f4-b57f-4225-9a4e-0b33ba4d0197 4.4.0-96-generic #119~14.04.1-Ubuntu SMP Wed Sep 13 08:40:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubectl
@obiyuta
Copy link
Author

@obiyuta obiyuta commented Jun 28, 2018

/sig cli

@k8s-ci-robot k8s-ci-robot added sig/cli and removed needs-sig labels Jun 28, 2018
@stephanwesten
Copy link

@stephanwesten stephanwesten commented Jun 28, 2018

I had the same problem on macos with the exact same versions.

Downgraded kubectl to v1.8.7 and it seems to solve the problem

wget https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/darwin/amd64/kubectl

It would be good to improve the error message, as a user you are clueless how to solve this issue.

@suryakanta87
Copy link

@suryakanta87 suryakanta87 commented Jun 28, 2018

I am also facing same problem please give solution.

No resources found.
Error from server (NotAcceptable): unknown (get nodes)

Thanks
srikant

@roaddemon
Copy link

@roaddemon roaddemon commented Jun 28, 2018

I just started getting this after upgrading kubectl to 1.11.0.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Downgrading to 1.8.7 to match my server version fixed my issue (same as stephanwesten previously mentioned)

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubectl
$ chmod +x ./kubectl
$ sudo mv ./kubectl /usr/local/bin/kubectl

@jesusvazquez
Copy link

@jesusvazquez jesusvazquez commented Jun 28, 2018

Yeah same happening here. Thanks @roaddemon for the workaround.

@dfjaimes
Copy link

@dfjaimes dfjaimes commented Jun 28, 2018

Same happened on Debian stable. Downgraded back to 1.10.5 and all is well

@liggitt
Copy link
Member

@liggitt liggitt commented Jun 28, 2018

kubectl supports +/- 1 version of skew against the API server. kubectl 1.11 defaults to requesting table output from the apiserver by sending the following accept header:

Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json

servers prior to 1.9 do not handle that accept header correctly (and return the "Not Acceptable" error, which kubectl displays). this was resolved in #50603

/close

@mhobotpplnet
Copy link

@mhobotpplnet mhobotpplnet commented Jun 29, 2018

Had same issues , for some reason I got to 1.11.0 version and that did not work..
MAC users try version 1.10.0 or replace the curl version with whatever you think it will work for you.

curl -LO curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/darwin/amd64/kubectl

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl
@maksymx
Copy link

@maksymx maksymx commented Jul 2, 2018

Thanks @roaddemon for workaround. I took version 1.10.5 instead of 1.8.7

@forestgagnon
Copy link

@forestgagnon forestgagnon commented Jul 3, 2018

Just got bit by this too and am having to pin the kubectl version, is this issue closed because it won't be fixed?

@liggitt
Copy link
Member

@liggitt liggitt commented Jul 3, 2018

is this issue closed because it won't be fixed?

Yes. 1.8 is outside the supported skew window for kubectl 1.11, and the server side bug causing this is fixed in 1.9+

@jasonyihk
Copy link

@jasonyihk jasonyihk commented Jul 12, 2018

On Mac:

brew info & brew switch will be much easier if you need to handle multiple cluster with different version.

@mprokopov
Copy link

@mprokopov mprokopov commented Jul 16, 2018

I've resolved the same issue by downgrading to 1.10.2

brew switch kubernetes-cli 1.10.2

@drkrishnan8109
Copy link

@drkrishnan8109 drkrishnan8109 commented Jul 22, 2018

Downgrading to version 1.10.0 fixed the issue for me.

@darthmachina
Copy link

@darthmachina darthmachina commented Jul 24, 2018

You can also add the -w switch or switch the output using -o. I had to do this when using kubectl on Azure in the cloud shell as I can't control the versions there. Not ideal but it will work.

@arani007
Copy link

@arani007 arani007 commented Jul 30, 2018

Thanks @roaddemon your solution worked like charm.

@pauldutheil
Copy link

@pauldutheil pauldutheil commented Aug 13, 2018

Got the same error with clusters running on 1.8.6

For macOs users, brew switch will only worked if you already have 1.10 installed before, if not, you can get it right from the commit

brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/54fc5aea2094d4d21e49383052fd2b975fbf6d6a/Formula/kubernetes-cli.rb
######################################################################## 100.0%
Warning: kubernetes-cli 1.11.2 is available and more recent than version 1.10.2.
==> Downloading https://homebrew.bintray.com/bottles/kubernetes-cli-1.10.2.el_capitan.bottle.tar.gz
…
brew info kubernetes-cli
kubernetes-cli: stable 1.11.2 (bottled), HEAD
Kubernetes command-line interface
https://kubernetes.io/
/usr/local/Cellar/kubernetes-cli/1.10.2 (178 files, 52.8MB)
  Built from source on 2018-08-13 at 15:44:36
/usr/local/Cellar/kubernetes-cli/1.11.0 (196 files, 53.8MB)
  Poured from bottle on 2018-07-12 at 13:49:59

then, switch to 1.10.2
brew switch kubernetes-cli 1.10.2

@lbrabus
Copy link

@lbrabus lbrabus commented Aug 14, 2018

thank you @pauldutheil this works for me now.

@xyalan
Copy link

@xyalan xyalan commented Aug 15, 2018

Thank you @pauldutheil

@wilsonsilva
Copy link

@wilsonsilva wilsonsilva commented Aug 20, 2018

Downgrading to v1.10.5 works well too.

@rafawalter
Copy link

@rafawalter rafawalter commented Aug 21, 2018

Downgraded to 1.10.5 as a workaround, looks fine
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/d09d97241b17a5e02a25fc51fc56e2a5de74501c/Formula/kubernetes-cli.rb
brew switch kubernetes-cli 1.10.5

Thanks @pauldutheil and @wilsonsilva

@jgschmitz
Copy link

@jgschmitz jgschmitz commented Sep 14, 2018

For me (version 1.11.3)
kubectl get pods returns
No resources found.
but kubectl get pods --namespace kube-system works
kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-4m6zx 1/1 Running 0 4h
coredns-78fcdf6894-s7zv7 1/1 Running 0 4h
etcd-sanchezk8s0 1/1 Running 9 4h
kube-apiserver-sanchezk8s0 1/1 Running 0 4h
kube-controller-manager-sanchezk8s0 1/1 Running 0 4h
kube-proxy-c7krq 1/1 Running 1 32m
kube-proxy-rbnkp 1/1 Running 0 4h
kube-proxy-xz8kq 1/1 Running 0 4h
kube-scheduler-sanchezk8s0 1/1 Running 0 4h
tiller-deploy-597c48f967-qzv4v 1/1 Running 0 3h
weave-net-2vlvj 2/2 Running 2 26m

@benbjurstrom
Copy link

@benbjurstrom benbjurstrom commented Sep 23, 2018

kubectl get pods --all-namespaces also works on 1.11.3

@jgschmitz
Copy link

@jgschmitz jgschmitz commented Sep 24, 2018

@benbjurstrom so the strangest thing now kubectl get pods is working also - like out of the blue

kubectl get po
NAME READY STATUS RESTARTS AGE
brazen-beetle-influxdb-7448f795cf-tznlw 1/1 Running 0 5d
eerie-meerkat-mongodb-674c7f7d67-5z2gr 0/1 Pending 0 6d
looping-giraffe-prometheus-alertmanager-6df9bcc54d-28mvh 0/2 Pending 0 6d
looping-giraffe-prometheus-kube-state-metrics-7ff8c9c789-5z4l5 1/1 Running 0 6d
looping-giraffe-prometheus-node-exporter-dqptv 1/1 Running 0 6d
looping-giraffe-prometheus-node-exporter-qt4sn 1/1 Running 0 6d
looping-giraffe-prometheus-pushgateway-6cb8687578-fpnwn 1/1 Running 0 6d
looping-giraffe-prometheus-server-69877f967c-7xn8f 0/2 Pending 0 6d

@yamlCase
Copy link

@yamlCase yamlCase commented Sep 28, 2018

Workaround I found on Azure (probably others as well) is to pass --server-print=false:

kubectl get componentstatuses --server-print=false

It appears to be related to a new "Server-side columns" feature introduced in 1.11.0. Here's a link to the reference: https://kubernetes.io/docs/reference/kubectl/overview/#server-side-columns

@darthmetu

edit: reference to azure issue: Azure/azure-cli#7440

@aasaru
Copy link

@aasaru aasaru commented Dec 9, 2018

I can confirm the workaround works also on Mac for getting pods:

$ kubectl get pods --server-print=false
While without --server-print the command gives an error:

$ kubectl get pods
Error from server (NotAcceptable): unknown (get pods)
@tubalu
Copy link

@tubalu tubalu commented Feb 16, 2019

Downgraded to 1.10.5 as a workaround, looks fine
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/d09d97241b17a5e02a25fc51fc56e2a5de74501c/Formula/kubernetes-cli.rb
brew switch kubernetes-cli 1.10.5

Thanks @pauldutheil and @wilsonsilva

thanks.

@hadooping
Copy link

@hadooping hadooping commented Feb 17, 2019

I downgraded to 10.10.5 and worked fine. Thanks @tubalu

@captn3m0
Copy link

@captn3m0 captn3m0 commented Mar 31, 2019

Since a lot of people on this thread are running a downgraded version of kubectl, please note that there is a new security release out for kubectl:

https://discuss.kubernetes.io/t/announce-security-release-of-kubernetes-kubectl-potential-directory-traversal-releases-1-11-9-1-12-7-1-13-5-and-1-14-0-cve-2019-1002101/5712

The issue is High severity and upgrading kubectl to Kubernetes 1.11.9, 1.12.7, 1.13.5, and 1.14.0 is encouraged to fix this issue.
Run kubectl version --client and if it does not say version 1.11.9, 1.12.7, 1.13.5, and 1.14.0 or newer you are running a vulnerable version.

@liggitt
Copy link
Member

@liggitt liggitt commented Mar 31, 2019

Note that you are only affected if you make use of the kubectl cp command

@javierg21
Copy link

@javierg21 javierg21 commented Sep 11, 2019

kubectl get pods --server-print=false
works on ubuntu, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet