Azure Monitor - InsightsMetrics - populate custom tag, or pod name

[sirkubax]

Azure Monitor - InsightsMetrics

Hello,

We need some way to filter metrics scraped from our services in Azure Kubernetes that are injected into InsightsMetrics.
It seems that labels/annotations from pods are droped, we did not find a way to populate pod name with metrics...

How should we plot a graph of requests per second per pod, in Azure Monitor, if we cannot filter out by a pod?

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-scrape-configuration
Azure/iotedge#6141

Example record from InsightsMetrics
12/3/2022, 10:32:00.000 AM aks-default-12500777-vmss000000 container.azm.ms/telegraf prometheus keycloak_response_errors 3 {"code":"500","container.azm.ms/clusterId":"/subscriptions/xxxxxx-5fa9-493e-9256-xxxxxx/resourceGroups/RG-dev/providers/Microsoft.ContainerService/managedClusters/aks-dev","container.azm.ms/clusterName":"aks-dev","hostName":"aks-default-12500777-vmss000000","method":"GET","resource":"realms,realms/center/protocol/openid-connect","scrapeUrl":"https://identity.dev.net/realms/center/metrics"} xxxxxx-9e53-4d70-9df1-xxxxxxxxxx

[sirkubax]

as I get it, Insights Metrics have limited Fields by default, I think a Computer 'field' could be a place to inject POD or container, or it could be placed into Tags

https://github.com/MicrosoftDocs/azure-reference-other/blob/main/azure-monitor-ref/tables/insightsmetrics.md

Maybe it is possible to write prometheus scrapper to inject such labels?

[sirkubax]

eh
found it

colleague definend URL scraping in Configmap in the section of
prometheus_data_collection_settings.node

so just use pod annotations and it would be picked up by prometheus_data_collection_settings.cluster