New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

Closed
obiyuta opened this Issue Jun 28, 2018 · 25 comments

Comments

Projects
None yet
@obiyuta

obiyuta commented Jun 28, 2018

/kind bug

What happened:

When getting nodes or pods using the command kubectl get pods kubectl get pods, the following output appears:

No resources found.
Error from server (NotAcceptable): unknown (get nodes)
No resources found.
Error from server (NotAcceptable): unknown (get pods)

What you expected to happen:

When running the command, kubectl get pods/nodes , with kubectl version 1.11.0, pods/nodes appear.

How to reproduce it (as minimally and precisely as possible):

  1. Run kubectl get pods using kubectl 1.11.0

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release):
  • Kernel:
Linux default-52e963f4-b57f-4225-9a4e-0b33ba4d0197 4.4.0-96-generic #119~14.04.1-Ubuntu SMP Wed Sep 13 08:40:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubectl
@obiyuta

This comment has been minimized.

obiyuta commented Jun 28, 2018

/sig cli

@k8s-ci-robot k8s-ci-robot added sig/cli and removed needs-sig labels Jun 28, 2018

@stephanwesten

This comment has been minimized.

stephanwesten commented Jun 28, 2018

I had the same problem on macos with the exact same versions.

Downgraded kubectl to v1.8.7 and it seems to solve the problem

wget https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/darwin/amd64/kubectl

It would be good to improve the error message, as a user you are clueless how to solve this issue.

@suryakanta87

This comment has been minimized.

suryakanta87 commented Jun 28, 2018

I am also facing same problem please give solution.

No resources found.
Error from server (NotAcceptable): unknown (get nodes)

Thanks
srikant

@roaddemon

This comment has been minimized.

roaddemon commented Jun 28, 2018

I just started getting this after upgrading kubectl to 1.11.0.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Downgrading to 1.8.7 to match my server version fixed my issue (same as stephanwesten previously mentioned)

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubectl
$ chmod +x ./kubectl
$ sudo mv ./kubectl /usr/local/bin/kubectl

@jesusvazquez

This comment has been minimized.

jesusvazquez commented Jun 28, 2018

Yeah same happening here. Thanks @roaddemon for the workaround.

@calamityman

This comment has been minimized.

calamityman commented Jun 28, 2018

Same happened on Debian stable. Downgraded back to 1.10.5 and all is well

@liggitt

This comment has been minimized.

Member

liggitt commented Jun 28, 2018

kubectl supports +/- 1 version of skew against the API server. kubectl 1.11 defaults to requesting table output from the apiserver by sending the following accept header:

Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json

servers prior to 1.9 do not handle that accept header correctly (and return the "Not Acceptable" error, which kubectl displays). this was resolved in #50603

/close

@mhobotpplnet

This comment has been minimized.

mhobotpplnet commented Jun 29, 2018

Had same issues , for some reason I got to 1.11.0 version and that did not work..
MAC users try version 1.10.0 or replace the curl version with whatever you think it will work for you.

curl -LO curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/darwin/amd64/kubectl

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl
@maksymx

This comment has been minimized.

maksymx commented Jul 2, 2018

Thanks @roaddemon for workaround. I took version 1.10.5 instead of 1.8.7

@forestgagnon

This comment has been minimized.

forestgagnon commented Jul 3, 2018

Just got bit by this too and am having to pin the kubectl version, is this issue closed because it won't be fixed?

@liggitt

This comment has been minimized.

Member

liggitt commented Jul 3, 2018

is this issue closed because it won't be fixed?

Yes. 1.8 is outside the supported skew window for kubectl 1.11, and the server side bug causing this is fixed in 1.9+

@jasonyihk

This comment has been minimized.

jasonyihk commented Jul 12, 2018

On Mac:

brew info & brew switch will be much easier if you need to handle multiple cluster with different version.

@mprokopov

This comment has been minimized.

mprokopov commented Jul 16, 2018

I've resolved the same issue by downgrading to 1.10.2

brew switch kubernetes-cli 1.10.2

@drkrishnan8109

This comment has been minimized.

drkrishnan8109 commented Jul 22, 2018

Downgrading to version 1.10.0 fixed the issue for me.

@darthmachina

This comment has been minimized.

darthmachina commented Jul 24, 2018

You can also add the -w switch or switch the output using -o. I had to do this when using kubectl on Azure in the cloud shell as I can't control the versions there. Not ideal but it will work.

@arani007

This comment has been minimized.

arani007 commented Jul 30, 2018

Thanks @roaddemon your solution worked like charm.

@pauldutheil

This comment has been minimized.

pauldutheil commented Aug 13, 2018

Got the same error with clusters running on 1.8.6

For macOs users, brew switch will only worked if you already have 1.10 installed before, if not, you can get it right from the commit

brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/54fc5aea2094d4d21e49383052fd2b975fbf6d6a/Formula/kubernetes-cli.rb
######################################################################## 100.0%
Warning: kubernetes-cli 1.11.2 is available and more recent than version 1.10.2.
==> Downloading https://homebrew.bintray.com/bottles/kubernetes-cli-1.10.2.el_capitan.bottle.tar.gz
…
brew info kubernetes-cli
kubernetes-cli: stable 1.11.2 (bottled), HEAD
Kubernetes command-line interface
https://kubernetes.io/
/usr/local/Cellar/kubernetes-cli/1.10.2 (178 files, 52.8MB)
  Built from source on 2018-08-13 at 15:44:36
/usr/local/Cellar/kubernetes-cli/1.11.0 (196 files, 53.8MB)
  Poured from bottle on 2018-07-12 at 13:49:59

then, switch to 1.10.2
brew switch kubernetes-cli 1.10.2

@lbrabus

This comment has been minimized.

lbrabus commented Aug 14, 2018

thank you @pauldutheil this works for me now.

@xyalan

This comment has been minimized.

xyalan commented Aug 15, 2018

Thank you @pauldutheil

@wilsonsilva

This comment has been minimized.

wilsonsilva commented Aug 20, 2018

Downgrading to v1.10.5 works well too.

@rafawalter

This comment has been minimized.

rafawalter commented Aug 21, 2018

Downgraded to 1.10.5 as a workaround, looks fine
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/d09d97241b17a5e02a25fc51fc56e2a5de74501c/Formula/kubernetes-cli.rb
brew switch kubernetes-cli 1.10.5

Thanks @pauldutheil and @wilsonsilva

@jgschmitz

This comment has been minimized.

jgschmitz commented Sep 14, 2018

For me (version 1.11.3)
kubectl get pods returns
No resources found.
but kubectl get pods --namespace kube-system works
kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-4m6zx 1/1 Running 0 4h
coredns-78fcdf6894-s7zv7 1/1 Running 0 4h
etcd-sanchezk8s0 1/1 Running 9 4h
kube-apiserver-sanchezk8s0 1/1 Running 0 4h
kube-controller-manager-sanchezk8s0 1/1 Running 0 4h
kube-proxy-c7krq 1/1 Running 1 32m
kube-proxy-rbnkp 1/1 Running 0 4h
kube-proxy-xz8kq 1/1 Running 0 4h
kube-scheduler-sanchezk8s0 1/1 Running 0 4h
tiller-deploy-597c48f967-qzv4v 1/1 Running 0 3h
weave-net-2vlvj 2/2 Running 2 26m

@benbjurstrom

This comment has been minimized.

benbjurstrom commented Sep 23, 2018

kubectl get pods --all-namespaces also works on 1.11.3

@jgschmitz

This comment has been minimized.

jgschmitz commented Sep 24, 2018

@benbjurstrom so the strangest thing now kubectl get pods is working also - like out of the blue

kubectl get po
NAME READY STATUS RESTARTS AGE
brazen-beetle-influxdb-7448f795cf-tznlw 1/1 Running 0 5d
eerie-meerkat-mongodb-674c7f7d67-5z2gr 0/1 Pending 0 6d
looping-giraffe-prometheus-alertmanager-6df9bcc54d-28mvh 0/2 Pending 0 6d
looping-giraffe-prometheus-kube-state-metrics-7ff8c9c789-5z4l5 1/1 Running 0 6d
looping-giraffe-prometheus-node-exporter-dqptv 1/1 Running 0 6d
looping-giraffe-prometheus-node-exporter-qt4sn 1/1 Running 0 6d
looping-giraffe-prometheus-pushgateway-6cb8687578-fpnwn 1/1 Running 0 6d
looping-giraffe-prometheus-server-69877f967c-7xn8f 0/2 Pending 0 6d

@yamlCase

This comment has been minimized.

yamlCase commented Sep 28, 2018

Workaround I found on Azure (probably others as well) is to pass --server-print=false:

kubectl get componentstatuses --server-print=false

It appears to be related to a new "Server-side columns" feature introduced in 1.11.0. Here's a link to the reference: https://kubernetes.io/docs/reference/kubectl/overview/#server-side-columns

@darthmetu

edit: reference to azure issue: Azure/azure-cli#7440

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment