Using Grafana/Prometheus to understanding 2i2c hub usage

[jmunroe]

Communities need better information on how their users are using the their hubs. We expose some information through Grafana dashboards but that tooling is not always how it should be used by Community Champions.

But even within the 2i2c team, it is sometimes unclear what information is actually available in Grafana and what it tells us.

I'd like to spend time this sprint improving our internal understanding of Grafana (with the intent of being able to share that knowledge with communities in a future sprint).

• Is using Grafana as a front-end application essential for building visualizations and downloading data? Is there any way of using a HTTP API to access that data from Python to load it into a Pandas dataframe?
• What Grafana dashboards should Partnerships being looking up to understand 'usage of hubs by our communities'?
• Can new Grafana dashboards be created?
• Is there data available in either GCP or AWS that can be queried through Grafana?
• Am I even using the term 'Grafana' correctly?

[jmunroe]

@choldgraf made this comment in Slack about using Prometheus/Grafana to understand the usage of our infrastructure

mostly I've wanted to use those graphs as a source of truth for our usage, number of hubs, etc, but it's too confusing for me to be able to use

In this sprint, I'd like to iterate on why this difficult and come up with a plan for what can be done today, and what we need to plan for the future development effort.

[jmunroe]

@sgibson91 -- I am considering asking questions about Grafana as part of this sprint. Is this a topic you'd be able to help me understand better? (Even if you are not an expert, just having the conversation around what is expected and what should not be expected would be valuable).

[choldgraf]

Just cross-linking the latest ask I made around the Global Usage Dashboard for Grafana:

2i2c-org/infrastructure#1888 (comment)

And for reference, here's a short video showing how to create the graph I want (more or less). This results in a plot that looks like "Weekly Active Users binned by cluster over the last 6 months". I don't think it'd be that hard to do it, but somebody just needs to make the proper adjustments.

CleanShot.2023-11-14.at.13.57.29.mp4

[sgibson91]

Can new Grafana dashboards be created?

Yes. The most robust way is to contribute them to this project: https://github.com/jupyterhub/grafana-dashboards
Once it is merged upstream, we can run our workflow here to deploy the dashboard to all our hubs: https://github.com/2i2c-org/infrastructure/actions/workflows/deploy-grafana-dashboards.yaml (This action overwrites any dashboards that were manually created in the grafana interface.)

This about covers the extent of my grafana knowledge.

[jmunroe]

No worries about grafana knowledge. It's an area that I keep on hitting as something I wish I knew more about. I'll pull you off this issue for the current sprint and let's prioritize other tasks first!

[jmunroe]

@yuvipanda has written up some notes get me started: https://hackmd.io/NllqOUfaTLCXcDQPipr4rg

• Provide guidance on how to access raw prometheus data infrastructure#3426

[colliand]

Recently, @yuvipanda wrote up a HackMD note "Access promethues data raw" in response to a request from @colliand. Yuvi's work should enable 2i2c's Parterships team to build notebook(s) that enable hub champions to get insights into hub usage patterns.

[yuvipanda]

Note that I got the same request from 3 people (@jmunroe, @colliand and Juliusz from LEAP) on the exact same week by coincidence :) 2i2c-org/infrastructure#3426 tracks the documentation itself, and will figure out where it lands.

[damianavila]

@jmunroe, @sgibson91, there is a chance to synergistically work with @GeorgianaElena (and I presume Yuvi) because she wants to move forward with some discussions/clarifications/docs/fixes about Grafana as part of the "Reduce Workload" goal last sprint starting tomorrow.