How are you doing metrics? #1713
-
|
Hi All I have since deployed ARC and have managed to get the self hosted runners up and running. We currently use Grafana internally so looking to see how I could use this with the runner deployment metrics Any help appreciated! |
Beta Was this translation helpful? Give feedback.
Replies: 10 comments 27 replies
-
|
You can use Prometheus to scrape metrics on action-controller-runner. Then you can connect Prometheus to Grafana The current version exposed following metrics.
In next release we will have following metrics too apart from above ones (PR already merged)
For some constraints we were using only the PercentBusy metric and not webhook metrics, we created a Grafana dashboard to track autoscaling metrics.
|
Beta Was this translation helpful? Give feedback.
-
|
@debugger24 Could you share the grafana json please |
Beta Was this translation helpful? Give feedback.
-
are these metrics usable if you're doing webhook based autoscaling, i.e. |
Beta Was this translation helpful? Give feedback.
-
|
@debugger24 can you provide the grafana dashboard json ? |
Beta Was this translation helpful? Give feedback.
-
|
If you could share the grafana dashboard json it would be very helpful , as it saves me half day 🫶 |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I just published this Grafana dashboard. |
Beta Was this translation helpful? Give feedback.
-
|
@debugger24 that's great, thanks for the help! set { Did you have to do any additional Config to get this working? Installed Prometheus Stack : |
Beta Was this translation helpful? Give feedback.
-
Can you share more details what errors are you getting ? |
Beta Was this translation helpful? Give feedback.
-
|
Not Seeing any specific error message just not seeing it picked up under service discovery - however github-actions-exporter is being picked up also using serviceMonitor. |
Beta Was this translation helpful? Give feedback.
-
|
Actions Metrics Yaml Output Github Actions Exporter Yaml Output |
Beta Was this translation helpful? Give feedback.
-
|
Tried to Disable the Proxy but leave ServiceMonitor enabled - Still not being picked up |
Beta Was this translation helpful? Give feedback.
-
|
In my case,
Config I used was In https://prometheus-community.github.io/helm-charts/ In https://actions-runner-controller.github.io/actions-runner-controller metrics:
serviceAnnotations: {}
serviceMonitor: false
serviceMonitorLabels: {}
port: 8443
proxy:
enabled: false
image:
repository: quay.io/brancz/kube-rbac-proxy
tag: v0.13.0Disabled If you use Prometheus Operator, and enable |
Beta Was this translation helpful? Give feedback.
-
|
Strangely when trying to hit the endpoint I am getting Connection Refused ? However when I enable port forwarding it works |
Beta Was this translation helpful? Give feedback.
-
|
Please i need some help in getting all the metrics provided by ARC. have been added to the new release ( runner-controller 0.22.0 ). So i upgraded my controllers to the new version but still just seeing only the following metrics We do use CDK to deploy the Helm resources in Eks and my configuration is as follows Chart-Version: actions-runner-controller-0.22.0 Please can someone help me figure this out? |
Beta Was this translation helpful? Give feedback.
-
|
Just be aware that Helm doesn't update CRDs, if that is necessary to have the metrics you need to update manually the CRD |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Moser-ss , can you please point me to a document on how to manually update the CRD ? I am also using the helm chart and only seeing half of the metrics coming up. |
Beta Was this translation helpful? Give feedback.
-
|
The CDRs are just normal Kubernetes resources so you can use kubectl apply and update CDR . The CDRs are located here charts/actions-runner-controller/crds Also the metrics server works with the input of the webhooks |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Moser-ss Thank you for your reply. I have the metrics annotation as mentioned in the troubleshooting document, and the prometheus available at cluster level
But the workflow metrics alone not coming up in the grafana, which is the metric I am really after, so I am kind of stuck. can you please help ! 
|
Beta Was this translation helpful? Give feedback.
-
|
@Moser-ss , I use fluxcd for deployment , so added crds upgrade in the template itself, but still only workflow metrics are missing though. any idea what else I might be missing ? TIA!! |
Beta Was this translation helpful? Give feedback.
-
|
Running chart version 0.22.0 I also can't see any of the new metrics. I haven't upgraded, I've just added some values to my Prometheus and ARC Helm charts: kube-prometheus: ARC: Pods: The I can't see those new metrics on the |
Beta Was this translation helpful? Give feedback.
-
|
I have the exact same problem. The pods are up but no metrics coming through |
Beta Was this translation helpful? Give feedback.
-
|
Also this issue is open from another user: #2799 I'm having a similar issue. I have ARC release update to the last version, CRDs are update also. But I just get 2 metrics
This is my configuration |
Beta Was this translation helpful? Give feedback.
-
|
This is the same for us. Everything on latest but only able to get these 2 metrics whilst using prometheus-community helm chart |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I am using v.027.5 image version and though I have jobs in queue, I dont find see these metrics reflecting the same, I am looking for a metric on how many jobs are in queue, so I can setup alert on that, if the queue is too big. |
Beta Was this translation helpful? Give feedback.
-
|
I can also only see two 2 metrics with everything latest? Any idea? horizontalrunnerautoscaler_spec_max_replicas |
Beta Was this translation helpful? Give feedback.
-
|
Anyone looking for a dashboard that works with the new metrics should check out this blog: https://www.kenmuse.com/blog/enabling-github-arc-metrics/ I have found that the cardinality on the gha_* metrics can be insane because they include runner_name (pod name) and runner_id (job id) essentially making one series per data point which prevents even using To clean up your metrics try using a service monitor configured like this. You'll need two, one for the controller, one for the listener (bulk of metrics). This is the one for the listener: |
Beta Was this translation helpful? Give feedback.
-
|
I haven't experienced metrics "disappearing" but we've only been running it for 4 days now. I'll restart to verify what happens. Prometheus counters are supposed to be resilient against restarts by their design. Prometheus should also be persisting your data so that could be a whole other issue (possibly misconfigured). I think you might be experiencing something similar to what I described about how the metrics have too high cardinality. Try a query (with inner subquery) like this: Here's the metrics config we use (which should be as described in the docs): |
Beta Was this translation helpful? Give feedback.
-
|
Whelp update. It seems like runner_id/runner_name is a known issue and the devs haven't fixed it. For now you can use the subquery I wrote to actually graph the metrics but the growth of the number of time series is enough to make me drop the metric entirely until a fix is made. We may try to deploy a copy of the code that removes those labels. |
Beta Was this translation helpful? Give feedback.
-
|
@jameshounshell , the query seems to fit my usecase, but what I am really trying to achieve is to get the total minutes/seconds of execution time taken by all workflows in a single value for a selected time window, like the usage minutes in GitHub billing. do you think it is possible with current structure of metric? please share your thoughts? thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I forked/edited the code to drop runner_id/runner_name and reduce the number of buckets. If you want to try running my version of the runner pull my repo and run this:
There's some silly checks that I couldn't figure out so I set the version to be the version number listed in the helm chart (currently ) otherwise the controller complains that the listener isn't the right version and won't deploy it. Add this to your values.yaml for the gha-runner-scale-set-controller helm chart |
Beta Was this translation helpful? Give feedback.
-
|
@rekha-prakash-maersk Here are the official prometheus docs for histogram manipulation https://prometheus.io/docs/practices/histograms/#count-and-sum-of-observations |
Beta Was this translation helpful? Give feedback.
You can use Prometheus to scrape metrics on action-controller-runner.
Then you can connect Prometheus to Grafana
The current version exposed following metrics.
horizontalrunnerautoscaler_spec_min_replicashorizontalrunnerautoscaler_spec_max_replicashorizontalrunnerautoscaler_status_desired_replicasIn next release we will have following metrics too apart from above ones (PR already merged)
horizontalrunnerautoscaler_replicas_desiredhorizontalrunnerautoscaler_runnershorizontalrunnerautoscaler_runners_registeredhorizontalrunnerautoscaler_runners_busyhorizontalrunnerautoscaler_terminating_busyhorizontalrunnerautoscaler_necessary_replicashorizontalrunnerautoscaler_workflow_runs_comp…