This repository has been archived by the owner. It is now read-only.

system:heapster ClusterRole lacking nodes/stats get privileges #1936

Closed
r0bj opened this Issue Jan 23, 2018 · 7 comments

Comments

Projects
None yet
8 participants
@r0bj

r0bj commented Jan 23, 2018

The system:heapster ClusterRole appears to be missing the ability to GET to nodes/stats resource.

heapster deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
        version: v6
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: gcr.io/google_containers/heapster:v1.5.0
        imagePullPolicy: Always
        command:
        - /heapster
        - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&insecure=true
        - --sink=influxdb:http://dev.os-metrics-db.service.poz-dev.consul:8086?withfields=true&db=k8s
        - --metric-resolution=30s

error messages from API server:

I0123 12:28:05.087431       1 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:heapster" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "get" resource "nodes/stats" named "dev-k8s-lb-p1" cluster-wide

error message from heapster:

E0123 12:28:35.033599       1 manager.go:101] Error in scraping containers from kubelet_summary:10.14.30.116:10250: request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=get, resource=nodes, subresource=stats)"

Issue is fixed after adding those lines to ClusterRole system:heapster:

- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get

kubernetes: 1.9.1
heapster: 1.5.0

@fejta-bot

This comment has been minimized.

fejta-bot commented Apr 23, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@r0bj

This comment has been minimized.

r0bj commented Apr 23, 2018

/remove-lifecycle stale

@OrlinVasilev

This comment has been minimized.

OrlinVasilev commented Aug 3, 2018

@r0bj - have started the kubelet with authorization-webhook true ? currently I'm facing the same issue which I cannot solve

@r0bj

This comment has been minimized.

r0bj commented Aug 3, 2018

Yes, kubelet is started with authorization-webhook true.

As a workaround I just created separate ClusterRole and use it for heapster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: heapster-custom-fix
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - deployments
  verbs:
  - get
  - list
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
@OrlinVasilev

This comment has been minimized.

OrlinVasilev commented Aug 7, 2018

thanks worked for me just starting the kubelet with --authentication-token-webhook=true and later on adding it to the kops config files.
thanks a lot hope will not hit the same on metrics server.

@ljfranklin

This comment has been minimized.

ljfranklin commented Oct 6, 2018

With K8S v1.12.0 I was able to get the Dashboard + Heapster working with this suggestion plus adding the verb create to the nodes/stats section to avoid a 403 error.

@Ascendance

This comment has been minimized.

Ascendance commented Oct 13, 2018

This fix should be implemented by default until metrics-server migration is complete. took me too long to figure out why heapster isn't working.

dghubble added a commit to poseidon/typhoon that referenced this issue Oct 19, 2018

Configure Heapster to source metrics from Kubelet authenticated API
* Heapster can now get nodes (i.e. kubelets) from the apiserver and
source metrics from the Kubelet authenticated API (10250) instead of
the Kubelet HTTP read-only API (10255)
* https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md
* Use the heapster service account token via Kubelet bearer token
authn/authz.
* Permit Heapster to skip CA verification. The CA cert does not contain
IP SANs and cannot since nodes get random IPs that aren't known upfront.
Heapster obtains the node list from the apiserver, so the risk of
spoofing a node is limited. For the same reason, Prometheus scrapes
skip CA verification for scraping Kubelet's provided by the apiserver.
* https://github.com/poseidon/typhoon/blob/v1.12.1/addons/prometheus/config.yaml#L68
* Create a heapster ClusterRole to work around the default Kubernetes
`system:heapster` ClusterRole lacking the proper GET `nodes/stats`
access. See kubernetes-retired/heapster#1936

dghubble added a commit to poseidon/typhoon that referenced this issue Oct 19, 2018

Configure Heapster to source metrics from Kubelet authenticated API
* Heapster can now get nodes (i.e. kubelets) from the apiserver and
source metrics from the Kubelet authenticated API (10250) instead of
the Kubelet HTTP read-only API (10255)
* https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md
* Use the heapster service account token via Kubelet bearer token
authn/authz.
* Permit Heapster to skip CA verification. The CA cert does not contain
IP SANs and cannot since nodes get random IPs that aren't known upfront.
Heapster obtains the node list from the apiserver, so the risk of
spoofing a node is limited. For the same reason, Prometheus scrapes
must skip CA verification for scraping Kubelet's provided by the apiserver.
* https://github.com/poseidon/typhoon/blob/v1.12.1/addons/prometheus/config.yaml#L68
* Create a heapster ClusterRole to work around the default Kubernetes
`system:heapster` ClusterRole lacking the proper GET `nodes/stats`
access. See kubernetes-retired/heapster#1936

@cblecker cblecker closed this Nov 30, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.