[Grafana][Observability] Embed Grafana dashboard panels into Ray dashboard #1278
Conversation
kevin85421
force-pushed
the
embed-grafana
branch
from
790218b
to
2e18904
Compare
kevin85421
changed the title
| "schemaVersion": 27, | ||
| "style": "dark", | ||
| "tags": [ | ||
| "rayVersion:2.5.0" |
There was a problem hiding this comment.
should this be 2.6.1?
It has some differences from the default dashboard generated automatically from 2.6.1
There was a problem hiding this comment.
Yes, that's why I mentioned that manually importing the dashboard, as stated in the "TODO: Note that importing the dashboard manually is not ideal", isn't ideal. We should find a way to import the Grafana dashboard automatically in the future. I added some sentences to describe it b94b5d6. Does the change make sense to you?
kevin85421
changed the title
|
cc @alanwguo would you mind reviewing this PR? Thanks! |
| * Note that we do not deploy Grafana in the head Pod, so we need to set both `RAY_GRAFANA_IFRAME_HOST` and `RAY_GRAFANA_HOST`. | ||
| `RAY_GRAFANA_HOST` is used by the head Pod to send health-check requests to Grafana in the backend. | ||
| `RAY_GRAFANA_IFRAME_HOST` is used by your browser to fetch the Grafana panels from the Grafana server rather than from the head Pod. | ||
| Because we forward the port of Grafana to `127.0.0.1:3000` in this example, we set `RAY_GRAFANA_IFRAME_HOST` to `http://127.0.0.1:3000`. |
There was a problem hiding this comment.
should we include instructions that if user does not want to port-forward, they should set up an ingress?
There was a problem hiding this comment.
We have instructions for setting up the ingress for the Ray dashboard. Setting up the ingress for Grafana seems to be beyond the scope of KubeRay. How about adding https://grafana.com/tutorials/run-grafana-behind-a-proxy/ to the doc?
There was a problem hiding this comment.
Add the link in 7362c58. Does it make sense to you? Thx!
…board (ray-project#1278) Embed Grafana dashboard panels into Ray dashboard
Why are these changes needed?
Replace
dashboard_default.jsonwithdefault_grafana_dashboard.json(generated by Ray 2.5.0)install/prometheus/install.sh&overrides.yaml: Pin thekube-prometheus-stackchart to version v48.2.1 and create the chart usingoverrides.yamlto configure Grafana. This will enable the embedding of Grafana dashboard panels into the Ray dashboard.ray-cluster.embed-grafana.yaml: An example YAML file to demonstrate embeddingRelated issue number
Closes #1020
Checks