Export metrics to prometheus

[lifeofmoo]

Hello,

Using the below as inspiration* I am trying to get my metrics sent to my Prom instance but I am not seeing anything in Prom even after running an successful workflow job.

Here are snippets of my values files:

actions-runner-controller

Enables the Metrics and Pod Annotations:

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "8080"

metrics:
  controllerManagerAddr: ":8080"
  listenerAddr: ":8080"
  listenerEndpoint: "/metrics"

gha-runner-scale-set

listenerTemplate:
  spec:
    containers:
      - name: listener
        securityContext:
          runAsUser: 1000
  metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8080"
      prometheus.io/path: "/metrics"

## listenerTemplate is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
# listenerTemplate:
#   spec:
#     containers:
#     # Use this section to append additional configuration to the listener container.
#     # If you change the name of the container, the configuration will not be applied to the listener,
#     # and it will be treated as a side-car container.
#     - name: listener
#       securityContext:
#         runAsUser: 1000
#     # Use this section to add the configuration of a side-car container.
#     # Comment it out or remove it if you don't need it.
#     # Spec for this container will be applied as is without any modifications.
#     - name: side-car
#       image: example-sidecar

kubectl get pod arc-runner-set-74467b46-listener -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"

kubectl get pod actions-runner-controller-gha-rs-controller-866fd898cc-2fb6v -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/default-container: manager
    prometheus.io/path: /metrics
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"

I've also verified that the metrics are accessible from both pods by doing using port-forwards and curling each localhost which return metrics.

Other info:

Helm chart version 0.9.3
Prom Version: 2.52.0
EKS 1.29

*Inspiration

https://www.kenmuse.com/blog/enabling-github-arc-metrics/
#3293
#3784

[Nastaliss]

I ended up adding a pod-monitor for each scale set

{{- with .Values.podMonitor -}}
{{- if .enabled }}
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: {{ $.Release.Name }}
spec:
  podMetricsEndpoints:
  - interval: {{ .interval }}
    port: {{ .port }}
    path: {{ .path }}
  selector:
    matchLabels:
      app.kubernetes.io/component: runner-scale-set-listener
      app.kubernetes.io/instance: {{ index $.Values "gha-runner-scale-set" "runnerScaleSetName" }}
{{- end -}}
{{- end }}

with the values :

podMonitor:
  enabled: true
  interval: 30s
  port: metrics
  path: /metrics

[lifeofmoo]

Interesting thanks. I've just done a bit more research and found this.

#3024

Shame the well written docs on enabling metrics don't mention this. There are quite a few tickets like mine asking them very same question.

[hikouki-gumo]

Your configuration looks correct.
How did you config Prometheus, static or service discovery? I guess somethings wrong with your Prom config, not ARC.

[lifeofmoo]

Hello @hikouki-gumo ,

I created two Service Monitors both "apps"

actions-runner-controller

kubectl get servicemonitor actions-runner-controller -n arc-systems -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: kube-prometheus-stack
  name: actions-runner-controller
  namespace: arc-systems
spec:
  endpoints:
  - interval: 15s
    path: /metrics
    port: "8080"
  namespaceSelector:
    matchNames:
    - arc-systems
  selector:
    matchLabels:
      app: actions-runner-controller

gha-runner-scale-set

kubectl get servicemonitor -n arc-runner-set gha-runner-scale-set -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: kube-prometheus-stack
  name: gha-runner-scale-set
  namespace: arc-runner-set
spec:
  endpoints:
  - interval: 15s
    path: /metrics
    port: "8080"
  namespaceSelector:
    matchNames:
    - arc-runner-set
  selector:
    matchLabels:
      app: gha-runner-scale-set

kubectl get prometheus kube-prometheus-stack-prometheus -n prometheus -o yaml | grep -B4 -A3 serviceMonitorSelector
    seccompProfile:
      type: RuntimeDefault
  serviceAccountName: kube-prometheus-stack-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack
  shards: 1

I believe service discovery is enabled because I am able to see these in the prom logs.

kubectl logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -n prometheus | grep "discovery manager" | grep arc
ts=2024-11-11T16:02:47.548Z caller=kubernetes.go:331 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/arc-systems/actions-runner-controller/0 msg="Using pod service account via in-cluster config"
ts=2024-11-11T16:02:47.549Z caller=kubernetes.go:331 level=info component="discovery manager scrape" discovery=kubernetes config=serviceMonitor/arc-runner-set/gha-runner-scale-set/0 msg="Using pod service account via in-cluster config"

However nothing is yet to land in Prom.

[lifeofmoo]

Still really stuck here -

I changed the ServiceMonitor to a PodMonitor as per @Nastaliss comments.

The good news is that the Service Discovery is now appearing. However the labels are being dropped.