Add custom labels to metrics in similar fashion as logs

[spedersen-emailage]

I had a look at the source code and it appears that metrics have four common labels and some unique, per-check labels. These do not appear to be customizable.

synthetic-monitoring-agent/internal/scraper/scraper.go

Line 369 in 24dfd70

metricLabels := []labelPair{

synthetic-monitoring-agent/internal/scraper/scraper.go

Line 328 in 24dfd70

// GrafanaCloud loki limits log entries to 15 labels.

Logs have seven common labels. Labels added to a check are passed to logs generated by the check in label_NAME format. For example, if I add env=dev to an HTTP check, logs produced by that check will have a label_env=dev label.

Would it be possible to extend the same functionality to metrics?

As is, we're unable to intelligently query any of the metrics generated by checks beyond what is available via the built-in labels. We can't group or filter by region, environment, etc.

Example metric labels:

probe_http_duration_seconds{}

Label	Description	Example Value
probe	probe name	grafana-probe-dev-ore
phase	check phase	connect
job	check name	check-name-dev-ore
instance	test endpoint	https://some-url.com/
config_version	internal reference	1678891414004722944

Example log labels:

Label	Description	Example Value
check_name	check type	http
instance	test endpoint	https://some-url.com/
job	check name	check-name-dev-ore
label_appname	custom label, from check config labels	some-app-name
label_env	custom label, from check config labels	dev
label_region	custom label, from check config labels	us-west-2
probe	probe name	grafana-probe-dev-ore
probe_success	probe success/failure	1
region	region, from probe config	us-west-2
source	source of log	synthetic-monitoring-agent

[spedersen-emailage]

Looks like I can do this with the sm_check_info metric. Got ahead of myself. I will take a look.

[spedersen-emailage]

Sorted!

avg by (label_region) (
  probe_duration_seconds{job=~"app-name-*", instance=~"https://app-name.*"}
    * on (job,instance,probe)
    group_left(label_env,label_appname,label_region)
    sm_check_info{label_env="$env",label_region=~"$region"}
      * on (instance,job,probe,config_version) 
      group_left
      probe_success{job=~"app-name-.*", instance=~"https://app-name.*"} > 0
)