Runner-scale-set metric cardinality

[hamishforbes]

The listener histograms gha_job_execution_duration_seconds and gha_job_startup_duration_seconds have extremely high cardinality.
After less than a week and ~800 jobs executed they are the top 2 highest cardinality metrics in my prometheus deployment, with more series than the next 4 metrics combined.

---

Specifically the runner_id and runner_name labels effectively results in a new unique histogram metric for every single job the scale set runs.

This would be a problem even if the metrics were just counters, but because they're histograms you actually get
(46 bucket metrics + sum + count) * 2 (execution+startup) + 2 (gha_started_jobs_total / gha_completed_jobs_total) = 96 new unique metrics for every executed job.
Only 1 of which is useful (sum) because actually you're only recording a single number, how many seconds it took for that unique job to run.

---

The job_workflow_ref is also problematic especially when running jobs on the PR event as the PR numeric ID is included and will create a unique job_workflow_ref label.

---

Finally job_name is also a potential issue, it is effectively 'user-controlled' and jobs using matrices can and will blow this up massively.

---

As these histograms also persist per-listener pod the number of samples scraped (and time/cpu required) also grows forever.
If your listener pod has been running for a week you'll still be scraping the ingesting the unique metric from the first job run on that scale set a week ago.

After about a week I was ingesting 80k samples per-scrape in around 0.5s.
Restarting all the listeners dropped that to 400 samples and around 10ms.

---

These labels should at minimum not be enabled by default as they will eventually break any prometheus deployment that doesn't take extra action to clean them up.

This graph shows the effect on cardinality of removing the various labels

Note this is just for gha_job_execution_duration_seconds_count, you have to multiply by 96 for the actual total number of unique series.

Partial workaround is to add a metric_relabeling config to your prometheus setup like:

  - source_labels: [__name__]
    regex: gha_job_(execution|startup)_duration_seconds_bucket
    action: drop

This will prevent scraping of the bucket and while it won't stop the sum and count metrics it should keep things under control, you'll probably want to periodically restart the listener pods as well.

[MPV]

Agree. 👍

Also, I'm less interested in the particular runner_name (and ID) and more interested in different runner groups and/or scale set names, which I don't see any labels for at the moment?

Would be really happy to see name/ID replaced into these (with a lot lower cardinality). 🙏

Use case example: measuring/comparing wait and execution times on different kinds of runner (scale sets), as they may use different images, nodes and resource requirements.

[MPV]

Another way (which I'm about to try out) for dropping just the labels with too high cardinality:

    metricRelabelings:
    - regex: runner_id
      action: labeldrop
    - regex: runner_name
      action: labeldrop

[nikola-jokic]

Hey @hamishforbes,

Thank you for reporting this issue! I agree this should be fixed ideally in the next or the release after that

[MPV]

Chiming in on the job_workflow_ref and job_name.

In my context we're mainly using reusable workflows, thus keeping cardinality quite low for these labels.
So in my context it's mainly the runner_id and runner_name which are most problematic.

I'd be fine if job ref/name were dropped of course, but I may see use cases for understanding things like:

• "Which of my reusable workflows take too long time to execute?"
• "How is execution times for my particular reusable workflow adjusting to changes in config for my runners/scale sets?"

[hamishforbes]

Chiming in on the job_workflow_ref and job_name.

In my context we're mainly using reusable workflows, thus keeping cardinality quite low for these labels.
So in my context it's mainly the runner_id and runner_name which are most problematic.

I'd be fine if job ref/name were dropped of course, but I may see use cases for understanding things like:

• "Which of my reusable workflows take too long time to execute?"
• "How is execution times for my particular reusable workflow adjusting to changes in config for my runners/scale sets?"

This is why they should probably be toggles

Job name is poor for us because we are using matrices to build multiple projects/architectures in the same workflows which means we get a unique "build ($project,$arch)" for each combination.

I'm using relabelling to regex out the workflow filename instead, same for extracting the repo, org and ref into separate labels
This allows me to have a dashboard that shows metrics for a specific repository

[MPV]

This is why they should probably be toggles

👍

Job name is poor for us because we are using matrices to build multiple projects/architectures in the same workflows which means we get a unique "build ($project,$arch)" for each combination.

Similar here.

I'm using relabelling to regex out the workflow filename instead, same for extracting the repo, org and ref into separate labels
This allows me to have a dashboard that shows metrics for a specific repository

Oh, would you be up for sharing? 😍

Can imagine using it for things like finding:

• slow/failing repos
• repos using deprecated workflows

Also curious to hear how you use it. 😇

[hamishforbes]

Oh, would you be up for sharing? 😍

We're in danger of going off topic but in case anyone else is running into issues with these metrics
I am using a prometheus-operator PodMonitor which currently looks like this:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: listener
  namespace: arc-system
spec:
  podMetricsEndpoints:
  - honorLabels: true
    metricRelabelings:
    # Drop high cardinality metrics
    - action: drop
      regex: gha_job_(execution|startup)_duration_seconds_(bucket|count)
      sourceLabels:
      - __name__
    # Rename these metrics with a __broken prefix
    # Use a recording rule to avg() them without `runner_id,runner_name` before shipping to Mimir
    # remote_write relabelling prevent shipping of the `__broken` prefixed metrics
    - action: replace
      regex: (gha_job_(execution|startup)_duration_seconds_sum)
      replacement: __broken_$1
      sourceLabels:
      - __name__
      targetLabel: __name__
    # Extract workflow filename
    - action: replace
      regex: .+\.github\/workflows\/(.+)@.+
      sourceLabels:
      - job_workflow_ref
      targetLabel: workflow
    # Extract repository name
    - action: replace
      regex: (.+)\/\.github.+
      sourceLabels:
      - job_workflow_ref
      targetLabel: repo
    # Extract git reference
    - action: replace
      regex: .+@(.+)
      sourceLabels:
      - job_workflow_ref
      targetLabel: ref
   # Remove job_workflow_ref
    - action: labeldrop
      regex: job_workflow_ref
    port: metrics
    relabelings:
    # Add the value of the `actions.github.com/scale-set-name` pod label as `scale_set` on the metric
    - sourceLabels:
      - __meta_kubernetes_pod_label_actions_github_com_scale_set_name
      targetLabel: scale_set
  selector:
    matchLabels:
      app.kubernetes.io/component: runner-scale-set-listener