Bind address for kube scheduler and controller manager

[adamhackl]

In order for Prometheus to properly scrape the health endpoints of kube-scheduler and kube-controller-manager, the --bind-address needs to be set to 0.0.0.0 or the control plane node IP. Clusters are currently deploying with the default of 127.0.0.1, which does not work in this case. Is there any way we can change the default setting or configure an override so clusters deploy this way instead of having to update the configuration post-deploy?

Thank you!

[mnaser]

👋 hi @adamhackl !

i think we can handle this with an extra cluster template label called kube_scheduler_bind_address and kube_controller_manager_bind_adddress which would default to 127.0.0.1 but then allowed to be overridden.

thoughts on this approach?

[adamhackl]

That would be a great solution for us. Could you also do something similar for the metricsBindAddress for kube-proxy?

Thank you!

[mnaser]

I think this needs to wait till #182 is done, without it, none of those ports will be exposed inside the cluster.

[Mazorius]

@mnaser that should be also implemented for etcd and kube-proxy … as they are also configured to 127.0.0.1 and can therefore not be scraped by Prometheus.

• etcd is also an control-plane manifest.
• kube-proxy is a daemon-set were the bind address must be changed.

etcd can be handled with this isssue. If kube-proxy should be tracked with a second can be the case.

#182 seems to be done already.

[mnaser]

Alright, so I went over the different components:

• kube-apiserver listens on the node IP by default, nothing to do.
• kube-controller-manager listens on 127.0.0.1:10257
• kube-scheduler listens on 127.0.0.1:10259
• kube-proxy listens on 127.0.0.1:10249
• etcd listens on 127.0.0.1:2381

from a security perspective, if we change those to 0.0.0.0 -- it only exposes it within the internal network the cluster lives in, as well as it is protected behind authorization which means someone has to be inside the network that the cluster lives in AND has access credentials that can pull in metrics / talk to metrics, and if you have that kind of credentials, you can probably spin up pods and do many other $things.

in this case, to verify, after updating kube-controller-manager:

root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl 10.0.0.163:10257
Client sent an HTTP request to an HTTPS server.
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl https://10.0.0.163:10257
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl -k https://10.0.0.163:10257
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl -k https://10.0.0.163:10257/metrics
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests#

..and for kube-scheduler:

root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl http://10.0.0.163:10259
Client sent an HTTP request to an HTTPS server.
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl https://10.0.0.163:10259
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl -k https://10.0.0.163:10259
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests# curl -k https://10.0.0.163:10259/metrics
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}
root@kube-9b7kp-control-plane-gmjzf-hfkcw:/host/etc/kubernetes/manifests#

for kube-proxy, it seems to be exposed, however, I'm happy with this since this is what AWS does for EKS: aws/containers-roadmap#657

for etcd, AKS seems to expose it out of the box as well: Azure/aks-engine#2989

I think I'm al in favour of sane defaults that work out of the box, most Kubernetes clusters will want/neeed kube-prometheus-stack and making them work out of the box is pretty key tbh

upon making the following changes:

• kube-controller-manager listens on 0.0.0.0:10257
• kube-scheduler listens on 0.0.0.0:10259
• kube-proxy listens on 0.0.0.0:10249
• etcd listens on 0.0.0.0:2381

i tried to deploy kube-prometheus-stack:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090

success... so i think we'll want to make the following adjustments:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
controllerManager:
  extraArgs:
    bind-address: 0.0.0.0
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
etcd:
  local:
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
metricsBindAddress: "0.0.0.0:10249"

once we have these set in place, users will get a much better experience out of the box and it will allow them to get something like kube-prometheus-stack working 100% out of the box. we're going to default to this all the time so no need to introduce more labels to configure things.

[okozachenko1203]

seems custom kubeproxyconfig is not supported in capi kubernetes-sigs/cluster-api#4512