New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to get kubernetes address: No kubernetes source found. #105

Open
Aleks3050 opened this Issue Aug 14, 2018 · 20 comments

Comments

Projects
None yet
10 participants
@Aleks3050

Aleks3050 commented Aug 14, 2018

Hi!
I try to deploy metrics-server to my kubernetes cluster(bare metal):
[root@node1 metrics-server]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

[root@node1 metrics-server]# kubectl get pods --namespace=kube-system --selector=k8s-app=metrics-server
NAME READY STATUS RESTARTS AGE
metrics-server-6cf87b9c8d-nvm9j 0/1 Error 8 16m
[root@node1 metrics-server]# kubectl logs metrics-server-6cf87b9c8d-nvm9j -n kube-system
I0814 08:12:57.748260 1 heapster.go:71] /metrics-server
I0814 08:12:57.748292 1 heapster.go:72] Metrics Server version v0.2.1
F0814 08:12:57.748300 1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

[root@node1 metrics-server]# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
creationTimestamp: 2018-08-14T07:40:59Z
name: v1beta1.metrics.k8s.io
resourceVersion: "13893882"
selfLink: /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io
uid: 63107df3-9f95-11e8-a53f-00259047efc9
spec:
caBundle: null
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
status:
conditions:

  • lastTransitionTime: 2018-08-14T07:40:59Z
    message: endpoints for service/metrics-server in "kube-system" have no addresses
    reason: MissingEndpoints
    status: "False"
    type: Available

Please helm with this issue!
Thank you!

@henley-regatta

This comment has been minimized.

Show comment
Hide comment
@henley-regatta

henley-regatta Aug 14, 2018

EDIT: This issue and the solution below is a duplicate of #97

When I pulled the project yesterday, I got an updated deploy/1.8+/metrics-server-deployment.yaml which is missing a command: stanza for the container: definition. I was able to fix by deleting the deployment, editing the file so the container definition reads:

containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

..then re-deploying. I suspect change a823af8 ("Refactor Makefile and update Dockerfile") is the cause, but I can't explain why (as I'd pulled the project onto another cluster after that change and had it work OK).

henley-regatta commented Aug 14, 2018

EDIT: This issue and the solution below is a duplicate of #97

When I pulled the project yesterday, I got an updated deploy/1.8+/metrics-server-deployment.yaml which is missing a command: stanza for the container: definition. I was able to fix by deleting the deployment, editing the file so the container definition reads:

containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

..then re-deploying. I suspect change a823af8 ("Refactor Makefile and update Dockerfile") is the cause, but I can't explain why (as I'd pulled the project onto another cluster after that change and had it work OK).

@Aleks3050

This comment has been minimized.

Show comment
Hide comment
@Aleks3050

Aleks3050 Aug 14, 2018

Ok! work fine after this changes but please tell me:
E0814 14:46:05.004234 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.85.94:10255): Get http://89.108.x.x:10255/stats/summary/: dial tcp 89.108.x.x:10255: getsockopt: connection refused

my kubelet on port 10250.
kubelet 1477 root 19u IPv4 26755 0t0 TCP 89.108.x,x:10250 (LISTEN)
how to change port for metrics-server?

maybe documentation? could not find

Aleks3050 commented Aug 14, 2018

Ok! work fine after this changes but please tell me:
E0814 14:46:05.004234 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.85.94:10255): Get http://89.108.x.x:10255/stats/summary/: dial tcp 89.108.x.x:10255: getsockopt: connection refused

my kubelet on port 10250.
kubelet 1477 root 19u IPv4 26755 0t0 TCP 89.108.x,x:10250 (LISTEN)
how to change port for metrics-server?

maybe documentation? could not find

@henley-regatta

This comment has been minimized.

Show comment
Hide comment
@henley-regatta

henley-regatta Aug 14, 2018

Aha, there's been a recent change and the read-only port used for metric gathering is disabled by default now. The quick hack to fix this is to add:

readOnlyPort: 10255

...to the bottom of /var/lib/kubelet/config.yaml and re-start kubelet (systemctl stop kubelet; sleep 10; systemctl start kubelet).

henley-regatta commented Aug 14, 2018

Aha, there's been a recent change and the read-only port used for metric gathering is disabled by default now. The quick hack to fix this is to add:

readOnlyPort: 10255

...to the bottom of /var/lib/kubelet/config.yaml and re-start kubelet (systemctl stop kubelet; sleep 10; systemctl start kubelet).

@DirectXMan12

This comment has been minimized.

Show comment
Hide comment
@DirectXMan12

DirectXMan12 Aug 14, 2018

Contributor

please don't do that. You should be using the authenticated port when possible. There's a reason the unauthenticated (read-only) port was disabled.

Contributor

DirectXMan12 commented Aug 14, 2018

please don't do that. You should be using the authenticated port when possible. There's a reason the unauthenticated (read-only) port was disabled.

@henley-regatta

This comment has been minimized.

Show comment
Hide comment
@henley-regatta

henley-regatta Aug 14, 2018

@DirectXMan12 Instructions for configuring metrics-server to do so would be greatly appreciated, because I looked but I did not find.

Nevermind. I see there's an example in #77 from @nasuku -

i.e
$ kubectl -n kube-system edit deploy metrics-server

look in section spec/template/spec/command for something like:
- --source=kubernetes
and replace that line with
- --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

henley-regatta commented Aug 14, 2018

@DirectXMan12 Instructions for configuring metrics-server to do so would be greatly appreciated, because I looked but I did not find.

Nevermind. I see there's an example in #77 from @nasuku -

i.e
$ kubectl -n kube-system edit deploy metrics-server

look in section spec/template/spec/command for something like:
- --source=kubernetes
and replace that line with
- --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

@DirectXMan12

This comment has been minimized.

Show comment
Hide comment
@DirectXMan12

DirectXMan12 Aug 14, 2018

Contributor

yep. That'll be the default (and only) option in the next released version (master today). Speaking of which, @kawych can we get a beta release pushed so people can try out the new version more?

Contributor

DirectXMan12 commented Aug 14, 2018

yep. That'll be the default (and only) option in the next released version (master today). Speaking of which, @kawych can we get a beta release pushed so people can try out the new version more?

@Aleks3050

This comment has been minimized.

Show comment
Hide comment
@Aleks3050

Aleks3050 Aug 14, 2018

hi!
@henley-regatta thank you for your answer.
I tried to install - --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

but see in logs:
E0814 19:14:05.017995 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.018016 1 summary.go:97] error while getting metrics summary from Kubelet node2(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.027822 1 summary.go:97] error while getting metrics summary from Kubelet node3(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

service account has been created:
[root@node1 metrics-server]# kubectl --namespace=kube-system describe serviceaccount metrics-server
Name: metrics-server
Namespace: kube-system
Labels:
Annotations:
Image pull secrets:
Mountable secrets: metrics-server-token-r2snk
Tokens: metrics-server-token-r2snk
Events:

Aleks3050 commented Aug 14, 2018

hi!
@henley-regatta thank you for your answer.
I tried to install - --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

but see in logs:
E0814 19:14:05.017995 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.018016 1 summary.go:97] error while getting metrics summary from Kubelet node2(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.027822 1 summary.go:97] error while getting metrics summary from Kubelet node3(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

service account has been created:
[root@node1 metrics-server]# kubectl --namespace=kube-system describe serviceaccount metrics-server
Name: metrics-server
Namespace: kube-system
Labels:
Annotations:
Image pull secrets:
Mountable secrets: metrics-server-token-r2snk
Tokens: metrics-server-token-r2snk
Events:

@Aleks3050

This comment has been minimized.

Show comment
Hide comment
@Aleks3050

Aleks3050 Aug 15, 2018

Hi!
Any updates?
Thank you!

Aleks3050 commented Aug 15, 2018

Hi!
Any updates?
Thank you!

@henley-regatta

This comment has been minimized.

Show comment
Hide comment
@henley-regatta

henley-regatta Aug 15, 2018

Sorry, can't help you. At a guess - and it is only a guess - you might need to re-run metrics-server/deploy/1.8+/auth-reader.yaml to (re-)create the role?

henley-regatta commented Aug 15, 2018

Sorry, can't help you. At a guess - and it is only a guess - you might need to re-run metrics-server/deploy/1.8+/auth-reader.yaml to (re-)create the role?

@Aleks3050

This comment has been minimized.

Show comment
Hide comment
@Aleks3050

Aleks3050 Aug 15, 2018

yes. i recreate role, but ... no effect.

Aleks3050 commented Aug 15, 2018

yes. i recreate role, but ... no effect.

@michelgokan

This comment has been minimized.

Show comment
Hide comment
@michelgokan

michelgokan Aug 16, 2018

I have the same issue.

michelgokan commented Aug 16, 2018

I have the same issue.

@cristianocasella

This comment has been minimized.

Show comment
Hide comment
@cristianocasella

cristianocasella commented Aug 17, 2018

same troubles

@danilorsilva

This comment has been minimized.

Show comment
Hide comment
@danilorsilva

danilorsilva Aug 17, 2018

Same problem here.

1 heapster.go:71] /metrics-server
1 heapster.go:72] Metrics Server version v0.2.1
1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

danilorsilva commented Aug 17, 2018

Same problem here.

1 heapster.go:71] /metrics-server
1 heapster.go:72] Metrics Server version v0.2.1
1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.
@nikhilkhanna

This comment has been minimized.

Show comment
Hide comment
@nikhilkhanna

nikhilkhanna Aug 19, 2018

I have the same issue as well, seems that deploy/1.8+/ is broken

nikhilkhanna commented Aug 19, 2018

I have the same issue as well, seems that deploy/1.8+/ is broken

@s-irvine

This comment has been minimized.

Show comment
Hide comment
@s-irvine

s-irvine Aug 20, 2018

FYI anyone getting:

heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

@unclok found a solution that seems to be working for a lot of people over in: #97 (comment)

s-irvine commented Aug 20, 2018

FYI anyone getting:

heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

@unclok found a solution that seems to be working for a lot of people over in: #97 (comment)

@Aleks3050

This comment has been minimized.

Show comment
Hide comment
@Aleks3050

Aleks3050 Aug 20, 2018

Hi!
At this moment error in pod metrics-server
E0814 19:14:05.017995 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.018016 1 summary.go:97] error while getting metrics summary from Kubelet node2(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.027822 1 summary.go:97] error while getting metrics summary from Kubelet node3(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

Aleks3050 commented Aug 20, 2018

Hi!
At this moment error in pod metrics-server
E0814 19:14:05.017995 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.018016 1 summary.go:97] error while getting metrics summary from Kubelet node2(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.027822 1 summary.go:97] error while getting metrics summary from Kubelet node3(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

@ggulati2

This comment has been minimized.

Show comment
Hide comment
@ggulati2

ggulati2 Aug 30, 2018

I am also getting unauthorized 401 error

ggulati2 commented Aug 30, 2018

I am also getting unauthorized 401 error

@Aleks3050

This comment has been minimized.

Show comment
Hide comment
@Aleks3050

Aleks3050 Aug 30, 2018

Hi!
I pull code and recreate metrics-server but error is same:
I0830 12:22:30.541833 1 serve.go:85] Serving securely on 0.0.0.0:443
E0830 12:23:05.017203 1 summary.go:97] error while getting metrics summary from Kubelet node3(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0830 12:23:05.018103 1 summary.go:97] error while getting metrics summary from Kubelet node1(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0830 12:23:05.023943 1 summary.go:97] error while getting metrics summary from Kubelet node2(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

Aleks3050 commented Aug 30, 2018

Hi!
I pull code and recreate metrics-server but error is same:
I0830 12:22:30.541833 1 serve.go:85] Serving securely on 0.0.0.0:443
E0830 12:23:05.017203 1 summary.go:97] error while getting metrics summary from Kubelet node3(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0830 12:23:05.018103 1 summary.go:97] error while getting metrics summary from Kubelet node1(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0830 12:23:05.023943 1 summary.go:97] error while getting metrics summary from Kubelet node2(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

@iakko

This comment has been minimized.

Show comment
Hide comment
@iakko

iakko Aug 30, 2018

I don't know if it's the same for minikube, but I've just installed it and cloned this repo.

Running the creation of Metrics Server, I got this error:

I0830 14:55:46.240497       1 heapster.go:71] /metrics-server
I0830 14:55:46.240548       1 heapster.go:72] Metrics Server version v0.2.1
F0830 14:55:46.240555       1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

Any hint?

iakko commented Aug 30, 2018

I don't know if it's the same for minikube, but I've just installed it and cloned this repo.

Running the creation of Metrics Server, I got this error:

I0830 14:55:46.240497       1 heapster.go:71] /metrics-server
I0830 14:55:46.240548       1 heapster.go:72] Metrics Server version v0.2.1
F0830 14:55:46.240555       1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

Any hint?

@ggulati2

This comment has been minimized.

Show comment
Hide comment
@ggulati2

ggulati2 Sep 4, 2018

Has anyone found a solution to fix it?

ggulati2 commented Sep 4, 2018

Has anyone found a solution to fix it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment