Need a dashboard

[aronchick]

We need a dashboard that walks our traces and produces the following things (daily):

• Number of jobs
• Length of each job running
• Spec of job (e.g. used GPU, amount of input data, amount of output data)
• Number of unique users submitting jobs
• Top users and jobs they submitted (in aggregate)
• Lots more... just start here :)

Design

• Internal dashboard, no auth needed
• a 'unique user' is basically the public key used by the cli to submit job (find out more)

Tasks

• Get stuff working in local mode & stuffed with workarounds
• Add a Prometheus systemd unit to terraform scripts
• Configure Prometheus to push to Grafana Cloud
• Add client-id to telemetry for Number of unique users submitting jobs
• Redeploy Production
• Configure dashboard(s) in Grafana
  • Connect data sources
  • Aggregate data (pending a Grafana support request re: cumulative sum)

Follow up: implement more metrics ---> #582

[lukemarsden]

Is this finished? Where's the dashboard?

[enricorotundo]

Waiting on PL infra team to provide us with a grafana cloud key. For some reason our (my & @aronchick ) accounts don't seem to be able to generate one.

Code wise it's all set up. We may want to add new metrics tho.

[enricorotundo]

Public dashboard: https://protocollabs.grafana.net/public-dashboards/9fe7717277e540a7be88f1e8e8339bd8?orgId=1

[enricorotundo]

Update: still busy configuring the Grafana dashboard to display the actual total sum of the metrics. These are counters and they do reset every now and then, so a simple sum is not enough. The closest I got to achieving that is with this Grafana query sum(increase(jobs_completed[720h])). WIP. Stay tuned

[enricorotundo]

Update: As per this thread, I've switched to using irate

Public link: https://protocollabs.grafana.net/public-dashboards/9fe7717277e540a7be88f1e8e8339bd8
Private link (for editing): https://protocollabs.grafana.net/d/x6I6OcG4z/bacalhau?orgId=1&from=now-7d&to=now

---

Follow up issue: #582 (Add more metrics to the codebase).

[enricorotundo]

Note Grafana cloud public dashboards have only alpha support - sometimes it's quite painful to have the public synced with the private dashboard :(