Grafana, Kiali, Jaeger links not showing up for EKS

[davidnuzik]

Version: master 2.2 (12/7/18)

What kind of request is this (question/bug/enhancement/feature request):
Bug

Steps to reproduce (least amount of steps as possible):

• Set up EKS cluster and enable monitoring
• Once it is ready notice there are no grafana links on the cluster monitoring screen. There are also none for projects, workloads, pods,containers, etc areas.
  Screenshot showing cluster monitoring page grafana links missing:
  

Result:
No grafana links when using EKS

Additional Information:
Based on discussions it seems like this might be a K8s issue. Only 1.12.3 seemed to work for showing the grafana links. This is likely an issue with the k8s proxy not being able to access grafana.

[davidnuzik]

GKS is not affected (Soumya tested)

[loganhz]

This is a backend issue. It works well in GKE and other hosted k8s

Now Rancher UI use the k8s proxy to access Grafana via /k8s/clusters/${cluster-id}/api/v1/namespaces/cattle-prometheus/services/http:access-grafana:80/proxy/

It looks like only kubernetes-dashboard is accessible with k8s proxy /k8s/clusters/${cluster-id}/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:443/proxy/

I tried to create a nginx workload, and create a service, and try to access nginx via /k8s/clusters/c-62mzm/api/v1/namespaces/default/services/http:abc:80/proxy/

it works well in a 1.12.3 cluster but it doesn’t work in v1.10.11-eks (edited)

[jiaqiluo]

The bug still exists on rancher: master 1/29
The Garafa icon is missing for both cluster and project level monitoring

[jiaqiluo]

The bug also exists in the HA install mode

Rancher: master on 2/26
Cluster: EKS

[vikas027]

I am too seeing this in Rancher v2.2.1 in an imported EKS cluster

[loganhz]

@vikas027 Current workaround is find access-grafana in system project service discovery tabs.
And look at what access-grafana does and create another node port service to expose Grafana UI via node port.

[loganhz]

@vikas027 you can add the service in UI

[Ilya-Kuchaev]

It doesn't work
Still get 503 when UI trys to access https://<rancher_url>/k8s/clusters/<cluser_id>/api/v1/namespaces/cattle-prometheus/services/http:access-grafana:80/proxy/api/search

Error: 'dial tcp 10.42.1.67:80: i/o timeout'
Trying to reach: 'http://10.42.1.67:80/api/search'

[loganhz]

Can you access the newly created node port service? #16968 (comment)

Something like node ip:node port

Is there a public ip on the node?

https://<rancher_url>/k8s/clusters/<cluser_id>/api/v1/namespaces/cattle-prometheus/services/http:access-grafana:80/proxy/api/search won't work in EKS

#16968 (comment)

[Ilya-Kuchaev]

Yes. I can access Grafana UI via the new Node Port. But the Rancher's Cluster UI still lacks its links. However, the metrics are functional

The cluster is a custom bare metal cluster with two nodes - 1 for management components and 1 for the worker.

[loganhz]

Ok, I thought you were using EKS.

Please don't do #16968 (comment) if it is a custom cluster. If you did, please revert it.

Can you deploy a busybox on each node to check if access-grafana:80 is accessable ?

[Ilya-Kuchaev]

Nope. But the issue is the same.
And yes, access-grafana:80 is accessible. Even via the full notation (access-grafana.cattle-prometheus.svc.cluster.local) from outside the cattle-prometheus namespace.

[loganhz]

So it's not just access-grafana service.
I think maybe for a simple nginx service, you still can not use https://<rancher_url>/k8s/clusters/<cluser_id>/api/v1/namespaces/${ns}/services/http:nginx:80/proxy/

I never hit it in a custom cluster. Is there anything special in your setup?

[Ilya-Kuchaev]

Yes. That's strange. Because I have two clusters configured via the same rancher. One custom cluster is fully functional and the second has this issue. Nothing special I believe. Except the second one had Prometheus installed via Helm before the upgrade to 2.1.

[mohag]

Fixing the CloudFormation templates you use to deploy the nodes is likely the most reliable solution. In the Amazon supplied one, search for "1025" and replace that with a number <= 80 to get this working

[davidnuzik]

Version: master-head (v2.3) (8/9/19)

This primarily changes the NodeSecurityGroup such that inbound port 80 is open now. If you have Rancher automatically set up the SG for you port 80 is now open. Reference rancher/kontainer-engine#175

I can now see Grafana links for monitoring and visit the links without any issues; I also enabled istio and ensured I can also access Istio Grafana, Kiali, Jaeger, Prometheus UIs without any issues.

SG before (rancher v2.2.7 used to check):

SG after the change:

Note: I checked the documentation and it already mentions inbound port 80 needed for nodes so no need for any documentation update. https://rancher.com/docs/rancher/v2.x/en/installation/requirements/ (networking tab)

[arbujsiddharth]

The bug still exists on rancher: master 1/29
The Garafa icon is missing for both cluster and project level monitoring

Hello All,
I am facing the same issue in Rancher 2.2.4 version.
I edited ControlPlaneSecurityGroup 1025-65535 to 0-65535 for WorkerSecurityGroup. But no luck. Has anyone found a solution? Appreciate help. Thanks.

[fmunteanu]

I confirm opening only port 80 between cluster and workers allows the Grafana links to be visible in Rancher 2.2.8 cluster section.

[chad-barensfeld-exa]

Experiencing the same issue with imported GKE cluster

[eramirez51]

I have same issue in GKE cluster

[davidnuzik]

@chad-barensfeld-exa and @eramirez51 can you both confirm port 80 is open? See #16968 (comment)

[billjh]

Had the same issue with imported GKE cluster (VPC native networking), fixed by adding a firewall rule to allow k8s api-server on master nodes (managed by GKE) to access pods

allow tcp (all)
source - master nodes cidr
target - pods cidr

reference: #19740 (comment)