Skip to content
This repository has been archived by the owner on Jan 5, 2022. It is now read-only.

/stats/summary returned HTTP status 401 Unauthorized #77

Open
casertap opened this issue Jan 18, 2019 · 13 comments
Open

/stats/summary returned HTTP status 401 Unauthorized #77

casertap opened this issue Jan 18, 2019 · 13 comments

Comments

@casertap
Copy link

Using this config in telegraf-ds

[[inputs.kubernetes]]
  bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  insecure_skip_verify = true
  url = "https://$HOSTIP:10250"

I get this error:

Jan 18 15:40:10 hosts-telegraf-ds-wsnbp hosts-telegraf-ds Error E! [inputs.kubernetes]: Error in plugin: https://172.23.47.179:10250/stats/summary returned HTTP status 401 Unauthorized

It seems that I am missing the serviceaccount token for telegraf to query the kubernetes endpoint.

I find it weird that the tick-charts did not create this serviceaccount automatically.

What am I missing? How can I make the inputs.kubernetes work?

@jackzampolin
Copy link
Contributor

@casertap when I wrote these charts serviceaccounts weren't a thing yet. You may need to add one to this chart.

@rawkode
Copy link
Contributor

rawkode commented Feb 5, 2019

@casertap I don't suppose you're running this on GKE?

@casertap
Copy link
Author

casertap commented Feb 8, 2019

@rawkode no I built my own kube cluster on aws using kops

@rawkode
Copy link
Contributor

rawkode commented Feb 8, 2019

@casertap Please ensure you have Webhook authentication enabled in your Kubelet configuration:

--authentication-token-webhook

@niklasember
Copy link

@rawkode I'm having this issue while running on GKE, any ideas?

Tried with --authentication-token-webhook on kubelet and have created a serviceaccount. Same config works on non-gke setup.

@rawkode
Copy link
Contributor

rawkode commented Feb 21, 2019

@niklasember GKE doesn't allow access to Kubelet on the host, you need to go through API Server

- kubernetes:
        url: "http://kubernetes.default.svc.cluster.local/v1/nodes/${HOSTIP}/proxy/metrics"
        bearer_token: "/var/run/secrets/kubernetes.io/serviceaccount/token"

@florianrusch
Copy link

@jackzampolin do you have an example how to configure the role for the service account?

@florianrusch
Copy link

I've found a solution that works for me:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tick-stack

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: metric-scanner-kubelet-api-admin
subjects:
- kind: ServiceAccount
  name: tick-stack
  namespace: tick
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io

@florianrusch
Copy link

We should update the telegraf-ds chart to also create this service-account automatically.

@rawkode
Copy link
Contributor

rawkode commented May 10, 2019

@florianrusch I agree. We're working on cleaning up our Helm charts and will be supporting / endorsing the charts in the official Helm repository very soon

@pbaderia01
Copy link

@florianrusch Would it be possible for you to list out the steps that you followed to get the service account working for you?

@florianrusch
Copy link

@piyush-insider I didn't test it again. But I've took the resources I've published before and just applied them to the cluster/namespace.

You can put the resources in a yaml-file and kubectl apply this file, or you can run this command:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tick-stack

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: metric-scanner-kubelet-api-admin
subjects:
- kind: ServiceAccount
  name: tick-stack
  namespace: tick
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io
EOF

@stanislav-zaprudskiy
Copy link

Apart from service account specification (which also requires #105) you'll need to provide adequate ClusterRole and ClusterRoleBinding.

For kubernetes plugin to work, assigning service account to system:kubelet-api-admin role (referenced above) is too much. In my case what was enough is something like below:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: telegraf
  namespace: monitoring
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: metrics-reader
rules:
- apiGroups: [""]
  resources: ["nodes/stats"]
  verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: telegraf-metrics-reader
subjects:
- kind: ServiceAccount
  name: telegraf
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: metrics-reader
  apiGroup: rbac.authorization.k8s.io

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants