Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

obiyuta · 2018-06-28T09:54:36Z

/kind bug

What happened:

When getting nodes or pods using the command kubectl get pods kubectl get pods, the following output appears:

No resources found.
Error from server (NotAcceptable): unknown (get nodes)

No resources found.
Error from server (NotAcceptable): unknown (get pods)

What you expected to happen:

When running the command, kubectl get pods/nodes , with kubectl version 1.11.0, pods/nodes appear.

How to reproduce it (as minimally and precisely as possible):

Run kubectl get pods using kubectl 1.11.0

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release):
Kernel:

Linux default-52e963f4-b57f-4225-9a4e-0b33ba4d0197 4.4.0-96-generic #119~14.04.1-Ubuntu SMP Wed Sep 13 08:40:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Install tools:

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubectl

The text was updated successfully, but these errors were encountered:

obiyuta · 2018-06-28T09:55:23Z

/sig cli

stephanwesten · 2018-06-28T11:30:11Z

I had the same problem on macos with the exact same versions.

Downgraded kubectl to v1.8.7 and it seems to solve the problem

wget https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/darwin/amd64/kubectl

It would be good to improve the error message, as a user you are clueless how to solve this issue.

suryakanta87 · 2018-06-28T17:14:27Z

I am also facing same problem please give solution.

No resources found.
Error from server (NotAcceptable): unknown (get nodes)

Thanks
srikant

roaddemon · 2018-06-28T18:00:09Z

I just started getting this after upgrading kubectl to 1.11.0.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Downgrading to 1.8.7 to match my server version fixed my issue (same as stephanwesten previously mentioned)

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubectl
$ chmod +x ./kubectl
$ sudo mv ./kubectl /usr/local/bin/kubectl

jesusvazquez · 2018-06-28T19:45:49Z

Yeah same happening here. Thanks @roaddemon for the workaround.

dfjaimes · 2018-06-28T20:32:04Z

Same happened on Debian stable. Downgraded back to 1.10.5 and all is well

liggitt · 2018-06-28T22:57:59Z

kubectl supports +/- 1 version of skew against the API server. kubectl 1.11 defaults to requesting table output from the apiserver by sending the following accept header:

Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json

servers prior to 1.9 do not handle that accept header correctly (and return the "Not Acceptable" error, which kubectl displays). this was resolved in #50603

/close

mhobotpplnet · 2018-06-29T10:49:14Z

Had same issues , for some reason I got to 1.11.0 version and that did not work..
MAC users try version 1.10.0 or replace the curl version with whatever you think it will work for you.

curl -LO curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/darwin/amd64/kubectl

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

maksymx · 2018-07-02T14:43:58Z

Thanks @roaddemon for workaround. I took version 1.10.5 instead of 1.8.7

forestgagnon · 2018-07-03T00:42:50Z

Just got bit by this too and am having to pin the kubectl version, is this issue closed because it won't be fixed?

liggitt · 2018-07-03T01:11:00Z

is this issue closed because it won't be fixed?

Yes. 1.8 is outside the supported skew window for kubectl 1.11, and the server side bug causing this is fixed in 1.9+

jasonyihk · 2018-07-12T06:11:48Z

On Mac:

brew info & brew switch will be much easier if you need to handle multiple cluster with different version.

mprokopov · 2018-07-16T13:41:31Z

I've resolved the same issue by downgrading to 1.10.2

brew switch kubernetes-cli 1.10.2

drkrishnan8109 · 2018-07-22T10:29:36Z

Downgrading to version 1.10.0 fixed the issue for me.

darthmachina · 2018-07-24T14:41:00Z

You can also add the -w switch or switch the output using -o. I had to do this when using kubectl on Azure in the cloud shell as I can't control the versions there. Not ideal but it will work.

arani007 · 2018-07-30T07:08:45Z

Thanks @roaddemon your solution worked like charm.

deliuth · 2018-08-13T20:14:20Z

Got the same error with clusters running on 1.8.6

For macOs users, brew switch will only worked if you already have 1.10 installed before, if not, you can get it right from the commit

brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/54fc5aea2094d4d21e49383052fd2b975fbf6d6a/Formula/kubernetes-cli.rb
######################################################################## 100.0%
Warning: kubernetes-cli 1.11.2 is available and more recent than version 1.10.2.
==> Downloading https://homebrew.bintray.com/bottles/kubernetes-cli-1.10.2.el_capitan.bottle.tar.gz
…

brew info kubernetes-cli
kubernetes-cli: stable 1.11.2 (bottled), HEAD
Kubernetes command-line interface
https://kubernetes.io/
/usr/local/Cellar/kubernetes-cli/1.10.2 (178 files, 52.8MB)
  Built from source on 2018-08-13 at 15:44:36
/usr/local/Cellar/kubernetes-cli/1.11.0 (196 files, 53.8MB)
  Poured from bottle on 2018-07-12 at 13:49:59

then, switch to 1.10.2
brew switch kubernetes-cli 1.10.2

lbrabus · 2018-08-14T09:01:09Z

thank you @pauldutheil this works for me now.

xyalan · 2018-08-15T06:36:32Z

Thank you @pauldutheil

wilsonsilva · 2018-08-20T08:57:04Z

Downgrading to v1.10.5 works well too.

rafawalter · 2018-08-21T13:06:04Z

Downgraded to 1.10.5 as a workaround, looks fine
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/d09d97241b17a5e02a25fc51fc56e2a5de74501c/Formula/kubernetes-cli.rb
brew switch kubernetes-cli 1.10.5

Thanks @pauldutheil and @wilsonsilva

jgschmitz · 2018-09-14T20:08:27Z

For me (version 1.11.3)
kubectl get pods returns
No resources found.
but kubectl get pods --namespace kube-system works
kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-4m6zx 1/1 Running 0 4h
coredns-78fcdf6894-s7zv7 1/1 Running 0 4h
etcd-sanchezk8s0 1/1 Running 9 4h
kube-apiserver-sanchezk8s0 1/1 Running 0 4h
kube-controller-manager-sanchezk8s0 1/1 Running 0 4h
kube-proxy-c7krq 1/1 Running 1 32m
kube-proxy-rbnkp 1/1 Running 0 4h
kube-proxy-xz8kq 1/1 Running 0 4h
kube-scheduler-sanchezk8s0 1/1 Running 0 4h
tiller-deploy-597c48f967-qzv4v 1/1 Running 0 3h
weave-net-2vlvj 2/2 Running 2 26m

benbjurstrom · 2018-09-23T16:00:36Z

kubectl get pods --all-namespaces also works on 1.11.3

jgschmitz · 2018-09-24T14:35:14Z

@benbjurstrom so the strangest thing now kubectl get pods is working also - like out of the blue

kubectl get po
NAME READY STATUS RESTARTS AGE
brazen-beetle-influxdb-7448f795cf-tznlw 1/1 Running 0 5d
eerie-meerkat-mongodb-674c7f7d67-5z2gr 0/1 Pending 0 6d
looping-giraffe-prometheus-alertmanager-6df9bcc54d-28mvh 0/2 Pending 0 6d
looping-giraffe-prometheus-kube-state-metrics-7ff8c9c789-5z4l5 1/1 Running 0 6d
looping-giraffe-prometheus-node-exporter-dqptv 1/1 Running 0 6d
looping-giraffe-prometheus-node-exporter-qt4sn 1/1 Running 0 6d
looping-giraffe-prometheus-pushgateway-6cb8687578-fpnwn 1/1 Running 0 6d
looping-giraffe-prometheus-server-69877f967c-7xn8f 0/2 Pending 0 6d

YAMLcase · 2018-09-28T13:34:39Z

Workaround I found on Azure (probably others as well) is to pass --server-print=false:

kubectl get componentstatuses --server-print=false

It appears to be related to a new "Server-side columns" feature introduced in 1.11.0. Here's a link to the reference: https://kubernetes.io/docs/reference/kubectl/overview/#server-side-columns

@darthmetu

edit: reference to azure issue: Azure/azure-cli#7440

aasaru · 2018-12-09T11:40:32Z

I can confirm the workaround works also on Mac for getting pods:

$ kubectl get pods --server-print=false
While without --server-print the command gives an error:

$ kubectl get pods
Error from server (NotAcceptable): unknown (get pods)

tubalu · 2019-02-16T20:51:35Z

Downgraded to 1.10.5 as a workaround, looks fine
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/d09d97241b17a5e02a25fc51fc56e2a5de74501c/Formula/kubernetes-cli.rb
brew switch kubernetes-cli 1.10.5

Thanks @pauldutheil and @wilsonsilva

thanks.

hadooping · 2019-02-17T17:33:55Z

I downgraded to 10.10.5 and worked fine. Thanks @tubalu

captn3m0 · 2019-03-31T11:27:37Z

Since a lot of people on this thread are running a downgraded version of kubectl, please note that there is a new security release out for kubectl:

https://discuss.kubernetes.io/t/announce-security-release-of-kubernetes-kubectl-potential-directory-traversal-releases-1-11-9-1-12-7-1-13-5-and-1-14-0-cve-2019-1002101/5712

The issue is High severity and upgrading kubectl to Kubernetes 1.11.9, 1.12.7, 1.13.5, and 1.14.0 is encouraged to fix this issue.
Run kubectl version --client and if it does not say version 1.11.9, 1.12.7, 1.13.5, and 1.14.0 or newer you are running a vulnerable version.

liggitt · 2019-03-31T12:08:11Z

Note that you are only affected if you make use of the kubectl cp command

liboyue23 · 2019-06-16T07:24:42Z

I try this reference is ok
https://www.benpickles.com/articles/72-downgrading-kubectl-with-homebrew

javierg21 · 2019-09-11T15:07:04Z

kubectl get pods --server-print=false
works on ubuntu, thanks

disznakul · 2021-02-15T16:20:09Z

Initial setup of kube on mac : error on viewing the pods.
Kubectl get pods
Unable to connect to the server: remote error: tls: handshake failure

I'm able to view kubectl config view (list of contexts)

k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. kind/bug Categorizes issue or PR as related to a bug. labels Jun 28, 2018

k8s-ci-robot added sig/cli Categorizes an issue or PR as relevant to SIG CLI. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 28, 2018

k8s-ci-robot assigned liggitt Jun 28, 2018

k8s-ci-robot closed this as completed Jun 28, 2018

AnthonyAmanse mentioned this issue Jul 2, 2018

Travis failing possibly due to kubectl version IBM/Kubernetes-container-service-GitLab-sample#82

Closed

filiphaftek mentioned this issue Jul 11, 2018

Upgraded default Kubernetes version russmedia/terraform-google-kubernetes-cluster#5

Merged

liggitt mentioned this issue Jul 16, 2018

kubectl get pods.v1beta1.metrics.k8s.io fails, but should succeed #65718

Closed

snpdev mentioned this issue Aug 12, 2018

Error from server (NotAcceptable): unknown (get nodes) Azure/AKS#598

Closed

jtuttle mentioned this issue Aug 21, 2018

Fix docker image deletion error cyberark/conjur#692

Merged

YAMLcase mentioned this issue Sep 28, 2018

Cloud Shell: 'kubectl get <object>' returns "Error from server (NotAcceptable): unknown" Azure/azure-cli#7440

Closed

jules-o mentioned this issue Oct 6, 2018

Seqr deploy failing with npm broadinstitute/seqr#536

Closed

bboreham mentioned this issue Sep 19, 2019

Launcher doesn't work with kubernetes <1.9 weaveworks/launcher#300

Closed

Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

obiyuta commented Jun 28, 2018

obiyuta commented Jun 28, 2018

stephanwesten commented Jun 28, 2018 •

edited

suryakanta87 commented Jun 28, 2018

roaddemon commented Jun 28, 2018

jesusvazquez commented Jun 28, 2018

dfjaimes commented Jun 28, 2018

liggitt commented Jun 28, 2018 •

edited

mhobotpplnet commented Jun 29, 2018 •

edited

maksymx commented Jul 2, 2018

forestgagnon commented Jul 3, 2018

liggitt commented Jul 3, 2018

jasonyihk commented Jul 12, 2018

mprokopov commented Jul 16, 2018

drkrishnan8109 commented Jul 22, 2018

darthmachina commented Jul 24, 2018

arani007 commented Jul 30, 2018

deliuth commented Aug 13, 2018

lbrabus commented Aug 14, 2018

xyalan commented Aug 15, 2018

wilsonsilva commented Aug 20, 2018

rafawalter commented Aug 21, 2018

jgschmitz commented Sep 14, 2018 •

edited

benbjurstrom commented Sep 23, 2018

jgschmitz commented Sep 24, 2018

YAMLcase commented Sep 28, 2018 •

edited

aasaru commented Dec 9, 2018

tubalu commented Feb 16, 2019

hadooping commented Feb 17, 2019

captn3m0 commented Mar 31, 2019

liggitt commented Mar 31, 2019

liboyue23 commented Jun 16, 2019

javierg21 commented Sep 11, 2019

disznakul commented Feb 15, 2021

Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

Kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4 #65575

Comments

obiyuta commented Jun 28, 2018

obiyuta commented Jun 28, 2018

stephanwesten commented Jun 28, 2018 • edited

suryakanta87 commented Jun 28, 2018

roaddemon commented Jun 28, 2018

jesusvazquez commented Jun 28, 2018

dfjaimes commented Jun 28, 2018

liggitt commented Jun 28, 2018 • edited

mhobotpplnet commented Jun 29, 2018 • edited

maksymx commented Jul 2, 2018

forestgagnon commented Jul 3, 2018

liggitt commented Jul 3, 2018

jasonyihk commented Jul 12, 2018

mprokopov commented Jul 16, 2018

drkrishnan8109 commented Jul 22, 2018

darthmachina commented Jul 24, 2018

arani007 commented Jul 30, 2018

deliuth commented Aug 13, 2018

lbrabus commented Aug 14, 2018

xyalan commented Aug 15, 2018

wilsonsilva commented Aug 20, 2018

rafawalter commented Aug 21, 2018

jgschmitz commented Sep 14, 2018 • edited

benbjurstrom commented Sep 23, 2018

jgschmitz commented Sep 24, 2018

YAMLcase commented Sep 28, 2018 • edited

aasaru commented Dec 9, 2018

tubalu commented Feb 16, 2019

hadooping commented Feb 17, 2019

captn3m0 commented Mar 31, 2019

liggitt commented Mar 31, 2019

liboyue23 commented Jun 16, 2019

javierg21 commented Sep 11, 2019

disznakul commented Feb 15, 2021

stephanwesten commented Jun 28, 2018 •

edited

liggitt commented Jun 28, 2018 •

edited

mhobotpplnet commented Jun 29, 2018 •

edited

jgschmitz commented Sep 14, 2018 •

edited

YAMLcase commented Sep 28, 2018 •

edited