New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubectl top node fails with heapster issue #46915

Closed
ompraash opened this Issue Jun 3, 2017 · 16 comments

Comments

Projects
None yet
10 participants
@ompraash
Copy link

ompraash commented Jun 3, 2017

We have a custom kube cluster. There is 1 Master and 2 Worker Node.

kubectl version is 1.5

Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:52:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Referred to : #30818 similar issue.

heapster version is v1.3.0-beta.1 in the default namespace.

kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
default       heapster-2326584968-qwqmf                  1/1       Running   0          22h
default       monitoring-grafana-1918260480-njf8s        1/1       Running   0          22h
default       monitoring-influxdb-201485718-mvkb5        1/1       Running   0          10d
kube-system   calico-node-56cps                          2/2       Running   0          10d
kube-system   calico-node-6pcjx                          2/2       Running   0          10d
kube-system   calico-node-hqj3k                          2/2       Running   0          10d
kube-system   calico-policy-controller-261675450-64c8j   1/1       Running   0          10d
kube-system   etcd-cni-kube-master-1                     1/1       Running   0          10d
kube-system   etcd-k8s-kube-master-1                     1/1       Running   0          10d
kube-system   kube-apiserver-kube-master-1               1/1       Running   0          10d
kube-system   kube-controller-manager-kube-master-1      1/1       Running   1          10d
kube-system   kube-dns-1.9-pbjb7                         3/3       Running   0          10d
kube-system   kube-proxy-4mzd6                           1/1       Running   0          10d
kube-system   kube-proxy-dqfmk                           1/1       Running   0          10d
kube-system   kube-scheduler-kube-master-1               1/1       Running   1          10d
kube-system   kubernetes-dashboard-cb62k                 1/1       Running   0          10d

cc @piosz

@k8s-merge-robot

This comment has been minimized.

Copy link
Contributor

k8s-merge-robot commented Jun 3, 2017

@ompraash There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here.

@xiangpengzhao

This comment has been minimized.

Copy link
Member

xiangpengzhao commented Jun 3, 2017

/sig cli

@xiangpengzhao

This comment has been minimized.

Copy link
Member

xiangpengzhao commented Jun 3, 2017

Since your heapster is not in kube-system namespace, try kubectl top node --heapster-namespace='default', please.

FYI: #44540

@ompraash

This comment has been minimized.

Copy link

ompraash commented Jun 3, 2017

the option is not supported with kubectl

kubectl top node --heapster-namespace='default'
Error: unknown flag: --heapster-namespace

@xiangpengzhao

This comment has been minimized.

Copy link
Member

xiangpengzhao commented Jun 3, 2017

can you see heapster related flags in kubectl top node -h? I'm not sure about which version introduced these flags.

@suyogbarve

This comment has been minimized.

Copy link
Contributor

suyogbarve commented Jun 3, 2017

@ompraash --heapster-namespace is available only after 1.6, I will recommend you to upgrade your client and server

@xiangpengzhao : 3d237ca

@xiangpengzhao

This comment has been minimized.

Copy link
Member

xiangpengzhao commented Jun 3, 2017

@suyogbarve 👍

@ompraash maybe you can try to deploy your heapster in kube-system namespace if you don't want to upgrade.

@suyogbarve

This comment has been minimized.

Copy link
Contributor

suyogbarve commented Jun 3, 2017

@ompraash You should always run heapster service and deployment in kube-system namespace, I believe even dashboard will not be able to display metrics if you run it in different namespace
If you run it in kube-system, you can also avoid necessity of passing --heapster-namespace as it defaults to kube-system

$ kubectl top node --heapster-namespace=kube-system
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-gce-us-central1-default-pool- 79m 7% 1685Mi 45%
gke-gce-us-central1-default-pool- 69m 6% 1680Mi 45%
gke-gce-us-central1-default-pool- 61m 6% 1869Mi 50%
$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-gce-us-central1-default-pool- 68m 6% 1676Mi 45%
gke-gce-us-central1-default-pool- 62m 6% 1864Mi 50%
gke-gce-us-central1-default-pool- 77m 7% 1685Mi 45%

@ompraash

This comment has been minimized.

Copy link

ompraash commented Jun 3, 2017

Thanks. @suyogbarve @xiangpengzhao
Yes dashboard wasn't displaying stats. Now after changing the namespace to kube-system the commands work and stats are generated.

kubectl top node
NAME            CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
kube-worker-1   375m         9%        5455Mi          34%       
kube-worker-2   165m         4%        5029Mi          21%       
kube-master-1   292m         7%        5328Mi          33% 

Is there any documentation which states heapster, dashboard, grafana & influxdb should always be in "kube-system" namespace. Examples from https://github.com/kubernetes/heapster/tree/master/deploy/kube-config shows in kube-system namespace.

@suyogbarve

This comment has been minimized.

Copy link
Contributor

suyogbarve commented Jun 3, 2017

@ompraash please close the issue , feel free to check back if you need anything.

@ompraash ompraash closed this Jun 3, 2017

@MilenkoM

This comment has been minimized.

Copy link

MilenkoM commented Jun 13, 2018

I have the same problem on my Ubuntu 16.04

kubectl top node
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)

If I try

kubectl get ns
NAME          STATUS    AGE
default       Active    5d
kube-public   Active    5d
kube-system   Active    5d
milenko@milenko-System-Product-Name:~$ kubectl top node --heapster-namespace=kube-system
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)

How to solve this problem?

@bfarayev

This comment has been minimized.

Copy link

bfarayev commented Jun 21, 2018

It can be because of client and server version mismatch @MilenkoM
Please check the output of

kubectl version

and I also noticed this issue while using GKE 1.10.4 and 1.10.2. As for now switched back to GKE 1.9.7

@fulvi0

This comment has been minimized.

Copy link

fulvi0 commented Jul 21, 2018

Hi folks,

I'm facing the same issue try to get stats from my nodes, I tried with the instructions above but without success.

Env info

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl get ns
NAME          STATUS    AGE
default       Active    3d
kube-public   Active    3d
kube-system   Active    3d
# cantonio @ srvlws01 in ~ [16:25:19]
$ kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

# cantonio @ srvlws01 in ~ [16:25:39] C:1
$ kubectl top node --heapster-scheme='http'
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

# cantonio @ srvlws01 in ~ [16:28:05] C:1
$ kubectl top node --heapster-namespace='default'
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

# cantonio @ srvlws01 in ~ [16:29:13] C:1
$ kubectl top node --heapster-namespace=kube-system
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
@ianjuma

This comment has been minimized.

Copy link

ianjuma commented Aug 8, 2018

I have the same issue.

Running 1.11 with a client on the same version.

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Looks like heapster crashes with a 503? And not related to an RBAC issue as earlier suspected.

logging error output: "Error: 'dial tcp 172.16.242.139:8082: i/o timeout'\nTrying to reach: 'http://172.16.242.139:8082/apis/metrics/v1alpha1/namespaces/hydra/pods?labelSelector='"
[[kubectl/v1.11.1 (linux/amd64) kubernetes/b1b2997] 10.100.100.100:57762]

GET /api/v1/namespaces/kubesystem/services/http:heapster:/proxy/apis/metrics/v1alpha1/namespaces/hydra/pods?labelSelector=

I0808 12:47:38.828533   14874 request.go:897] Response Body: Error: 'dial tcp 172.16.242.139:8082: i/o timeout'
Trying to reach: 'http://172.16.242.139:8082/apis/metrics/v1alpha1/namespaces/hydra/pods?labelSelector='
I0808 12:47:38.829225   14874 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server is currently unable to handle the request (get services http:heapster:)",
  "reason": "ServiceUnavailable",
  "details": {
    "name": "http:heapster:",
    "kind": "services",
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "Error: 'dial tcp 172.16.242.139:8082: i/o timeout'\nTrying to reach: 'http://172.16.242.139:8082/apis/metrics/v1alpha1/namespaces/hydra/pods?labelSelector='"
      }
    ]
  },
  "code": 503
}]
F0808 12:47:38.829335   14874 helpers.go:119] Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)

Looks like it's not able to work through the apiserver internal proxy.

curl -X GET  --cacert /var/lib/kubernetes/ca.pem  \ 
'https://10.100.100.100:6443/api/v1/namespaces/kubesystem/services/heapster/apis/metrics/v1alpha1/namespaces/hydra/pods?labelSelector='  \
 -H 'authorization: Bearer <token>'
@ianjuma

This comment has been minimized.

Copy link

ianjuma commented Aug 8, 2018

Appears to be linked to issue #56226 and #55238 ; where for aggregated services, pod/service networking is needed on the apiserver for this to work. Running kube-proxy and adding pod connectivity fixed this for me.
Error 503 is also misleading here. Not sure if this is the apiserver/ heapster.

@nsidhaye

This comment has been minimized.

Copy link

nsidhaye commented Oct 15, 2018

I am also facing extact same issue mentioned by @ianjuma

kubectl top node --heapster-namespace=kube-system
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)

I am using 1.10.1

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Should we reopen this issue or create new one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment