New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`kubectl get cs`: incorrect hard coded master component locations #27607

Open
DutchakVitaliy opened this Issue Jun 17, 2016 · 11 comments

Comments

Projects
None yet
@DutchakVitaliy

DutchakVitaliy commented Jun 17, 2016

I am trying to run controller-manager and scheduler on custom private ip.
But kubectl get cs trying to check health on 127.0.0.1 not on ip where they bind.

Kubernetes version: 1.2.4

kube-controller-manager.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-controller-manager
    image: 'quay.io/coreos/hyperkube:v1.2.4_coreos.0'
    command:
        - /hyperkube
        - controller-manager
        - --address=10.0.0.10
        - --master=http://127.0.0.1:8080
        - --leader-elect=true
        - --service-account-private-key-file=/etc/kubernetes/ssl/server-key.pem
        - --root-ca-file=/etc/kubernetes/ssl/ca.pem
        - --pod-eviction-timeout=30s
    livenessProbe:
      httpGet:
        host: 10.0.0.10
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true

    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates

kube-scheduler.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: quay.io/coreos/hyperkube:v1.2.4_coreos.0
    command:
    - /hyperkube
    - scheduler
    - --master=http://127.0.0.1:8080
    - --leader-elect=true
    - --address=10.0.0.10
    livenessProbe:
      httpGet:
        host: 10.0.0.10
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1

kubectl get cs

NAME                 STATUS      MESSAGE                                                                                                   ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connection refused                          
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connection refused                          

I think problem is here:
pkg/master/master.go

...
func (m *Master) getServersToValidate(c *Config) map[string]apiserver.Server {
    serversToValidate := map[string]apiserver.Server{
        "controller-manager": {Addr: "127.0.0.1", Port: ports.ControllerManagerPort, Path: "/healthz"},
        "scheduler":          {Addr: "127.0.0.1", Port: ports.SchedulerPort, Path: "/healthz"},
    }
...
@adohe

This comment has been minimized.

Member

adohe commented Jun 17, 2016

A possible mechanism is controller-manager and scheduler can report their ips to apiserver.

@lavalamp

This comment has been minimized.

Member

lavalamp commented Jun 18, 2016

There's a big component registration proposal laying around the repo somewhere.

@lavalamp lavalamp changed the title from `kubectl get cs` problem with controller-manager and scheduler served on custom ip to `kubectl get cs`: incorrect hard coded master component locations Jun 18, 2016

@adohe

This comment has been minimized.

Member

adohe commented Jun 18, 2016

I would take a look. Thanks @lavalamp

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Jun 24, 2016

See #13216

@Bregor

This comment has been minimized.

Bregor commented Oct 1, 2016

BTW, running scheduler with --address=127.0.0.1 is impossible too.
Is there any other way to hide ports 1025* from outside world?

@jralmaraz

This comment has been minimized.

jralmaraz commented Apr 10, 2017

Just spinned up a Kubernetes cluster using Rancher and I am getting the same unhealthy status. Rancher ran by default on 127.0.0.1. Just reporting to see if this is specific to Rancher on can be affecting someone else.

kubectl get cs
NAME STATUS MESSAGE
ERROR
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection re
fused
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection re
fused
etcd-0 Healthy {"health": "true"}

@fejta-bot

This comment has been minimized.

fejta-bot commented Dec 23, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@githubixx

This comment has been minimized.

githubixx commented Jan 6, 2018

That's really sad that nobody takes care of this issue. It's really an annoying bug if you run Kubernetes e.g. on bare metal. The bug still exists with k8s v1.9.1... Is it really that hard to fix? I would try myself but my Go experience isn't sufficient. But I've a test env. and could help debugging if this would help.

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Jan 22, 2018

/remove-lifecycle stale
/lifecycle frozen

@hhoover

This comment has been minimized.

Contributor

hhoover commented Nov 29, 2018

Using IPv6 with 1.12, setting proper bind addresses works for the controller manager and scheduler pods. Works great. However, when running this particular command (get cs) the 127.0.0.1 is hard coded, so it shows those components as being unhealthy. This call should actually use the address these services are bound to, not 127.0.0.1.

"controller-manager": {Addr: "127.0.0.1", Port: ports.InsecureKubeControllerManagerPort, Path: "/healthz"},
"scheduler": {Addr: "127.0.0.1", Port: ports.InsecureSchedulerPort, Path: "/healthz"},

@mauilion

This comment has been minimized.

Contributor

mauilion commented Nov 30, 2018

Arguably, it's better to bind services like controller-manager and scheduler to 127.0.0.1 this restricts things that aren't sharing that stack to access it.

But I need access to the /metrics endpoint!!

Enter kube-rbac-proxy or any other proxy really. This enables you to deploy a solution that would require that callers to that /metrics endpoint come proper with a service account that allows access.

INFO: https://github.com/brancz/kube-rbac-proxy

In this way you can expose the metrics endpoints for these tools and not expose other potentially interesting information to the unauthenticated masses like:

curl http://127.0.0.1:10252/configz
or curl http://127.0.0.1:10252/debug/pprof
etc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment