Could not connect to AWS EKS cluster

[tonimontero]

Hello, following the setup instructions, i'm not being able to connect our grafana to the AWS EKS cluster. the following message is given:

Tried to deploy cluster node exporters manually with given json, but still not able to connect grafana with eks. There are some rbac files that i should also deploy?

Thanks in advance.

[dotanalon]

managed to solve it?

[tonimontero]

managed to solve it?

Still not.

[mmingorance-dh]

👍
I'm also having this issue.
The problem comes that in EKS we can not have Static Password File as we do in Kops and also, EKS does not generate a client certificate and a key certificate. So, none of the current authentications solutions supported by this plugin works for us.

Is there a change that you can implement ServiceAccount authentication for this plugin? In this way we will be able to deploy this plugin on EKS.

[tonimontero]

👍
I'm also having this issue.
The problem comes that in EKS we can not have Static Password File as we do in Kops and also, EKS does not generate a client certificate and a key certificate. So, none of the current authentications solutions supported by this plugin works for us.

Is there a change that you can implement ServiceAccount authentication for this plugin? In this way we will be able to deploy this plugin on EKS.

That would be nice. Right now as a workarround i have a second grafana+prometheus stack inside the EKS cluster.

[mmingorance-dh]

@tonimontero We also have everything running inside the EKS cluster(prometheus as a deployment and grafana too)
How are you setting this workaround? It's very important for us. Would be great if you could help us!

[tonimontero]

@tonimontero We also have everything running inside the EKS cluster(prometheus as a deployment and grafana too)
How are you setting this workaround? It's very important for us. Would be great if you could help us!

What you have said, is exactly what we are doing... Isn't working for you? Maybe this links could help you:

https://sysdig.com/blog/kubernetes-monitoring-prometheus-operator-part3/

[mmingorance-dh]

@tonimontero are you creating your own dashboards or are you using the Grafana K8s plugin to do so?
What I'm trying to do is to use the Grafana K8s plugin to automatically create the dashboards and import the right metrics (as it does on my K8s cluster running with Kops when I click on deploy), but since EKS does not let me connect the plugin with the Kubernetes API, I can't get this working.

Another thing I thought it was to create my own dashboards based on the ones that this plugin creates.
How are exactly doing then?

Thanks for your help!

[adaniline-traderev]

If we deployed prometheus operator with kube-state-metrics and node-exporter, can't we just use the dashboards with the existing Prometheus data source? What is the point in configuring a cluster and a new data source, since no new deployments are required?

EKS authenticates with STS tokens based on AWS credentials (aws-iam-authenticator) - can support for this be added in the future?

[toughrogrammer]

I also cannot use kubernetes plugin because there is no client certificate and key of EKS cluster

[txynidakis]

Hi, has anyone managed to get the Kubernetes-app working with EKS?

Can't find any clear references anywhere.

[aarongorka]

@txynidakis I have got it working, but I would not bother with this app if I had to do it again. Half of the metric names are now wrong and making it work with EKS involves removing all the Kubernetes datasources anyway. You're probably better off just installing Prometheus/node-exporter/api-state-metrics yourself and using a dashboard from https://grafana.com/grafana/dashboards

[DimitrijeManic]

@aarongorka I suggest you go with https://github.com/kubernetes-monitoring/kubernetes-mixin. Also have a look at https://github.com/helm/charts/tree/master/stable/prometheus-operator which comes with everything working right out of the box

[txynidakis]

Thanks @aarongorka & @DimitrijeManic We've managed to get Prometheus-Operator going, we just wanted to see if the plug-in was worthwhile or worth the effort.

[CrusaderX]

As a workaround you can add sidecar container into grafana deployment with kubectl preinstalled and command kubectl proxy (do not forget to create/delegate properly RBAC permissions)

[JeroenAP]

Hi @CrusaderX, could you give some info on this? What permissions would be required? and what do you mean with sidecar container?

[CrusaderX]

@JeroenAP Something like this:

spec:
  serviceAccountName: myServiceAccountName
  containers:
  - name: kubectl
    image: bitnami/kubectl
    command:
      - kubectl
      - proxy
  - name: grafana:latest
    ...

where myServiceAccountName is a serviceAccount name with permissions which needed for Kubernetes-app. As a temporary solution for testing you can create ClusterRoleBinding with cluster-admin role:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: default-rbac
subjects:
  - kind: ServiceAccount
    name: myServiceAccountName
    namespace: myNamespace
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

In grafana Kubernetes-app plugin settings URL will be http://localhost:8001 without any additional settings.

[saikatharryc]

@CrusaderX I was trying the same, just the I still receive 403, (if I do curl http://localhost:8001 from the grafana container)
I have checked the sidecar, which does not have a Kube config.
I can add the kubeconfig there, but it has the aws-iam-authenticator thingy,

did you get it working with EKS somehow?

[saikatharryc]

Nevermind, I got it working. seems like I had to give a cluster-admin role. or maybe some type of role, which will allow you to get/list/view pretty much everything.