Values not showing up at /metrics endpoint

[deggja]

Hey guys,

I am testing your log analytics exporter (great job by the way). I am running some queries locally with the debug and trace flags and I see that my queries are fetching the right amount of results when I compare and run the same query in Log Analytics. However, these results are not shown on the /probe endpoint.

The stack trace showing that the query fetched results:

EBU[0004]/home/runner/work/azure-loganalytics-exporter/azure-loganalytics-exporter/loganalytics/prober.go:358 loganalytics.(*LogAnalyticsProber).executeQueries fetched 4 results metric=azure_metrics_loganalytics_exporter_missing_heartbeat module= results=4

The following config reproduces this issue:

- metric: azure_metrics_loganalytics_exporter_missing_heartbeat
    query: |-
      Heartbeat 
      | summarize LastHeartbeat=max(TimeGenerated) by Computer
      | where LastHeartbeat < ago(5h)
      | where Computer !contains_cs "avd"
    fields:
      - name: Computer
        type: id

Do you guys have any idea as to why this is happening? Is it because of a missing type: value field? Could not find any documentation on this so its mostly been trial and error, but I have managed to get it to work with multiple other queries.

Every pointer in the right direction is appreciated. Thanks.

[deggja]

I just want to add that there is not necessarily any need to resolve this specific problem. It would be equally or more useful with some hints as to how I should go about writing my configuration to get the best results.

I've already tested this and it currently is running as expected on Kubernetes, however, I cant seem to completely understand some of these queries not working.. so yeah thats why I created this issue.

[mblaschke]

is it possible to post the whole config file or is that anything? do you specify a timerange?

you're specifing the workspace in the app arguments, right?

[deggja]

is it possible to post the whole config file or is that anything? do you specify a timerange?

you're specifing the workspace in the app arguments, right?

Hi, thanks for replying @mblaschke 🚀

for the local test I am passing the workspace in the command when running the binary like such:

./azure-loganalytics-exporter --loganalytics.workspace=11111-22222-3333-4444 -c config.yml --log.trace --log.debug

This is the entire config (the first two queries work fine and give me the result I want, the heartbeat one fetches four results but shows no metrics at the /metrics endpoint):

queries:
  - metric: azure_metrics_loganalytics_exporter_available_memory
    query: |-
      // Virtual Machine available memory. 
      InsightsMetrics
      | where TimeGenerated > ago(1h)
      | where Origin == "vm.azm.ms"
      | where Namespace == "Memory"
      | where Name == "AvailableMB"
      | summarize avg(Val) by bin(TimeGenerated, 5m), Computer
      | render timechart 
    # timespan: PT1H
    fields:
      - name: avg_Val
        type: value
      - name: Computer
        type: id
    defaultField:
      type: ignore
  # AVAILABLE MEMORY IN PERCENT
  - metric: azure_metrics_loganalytics_exporter_available_memory_percent
    query: |-
      InsightsMetrics
      | where Origin == "vm.azm.ms"
      | where Computer !contains_cs "dev"
      | where Namespace == "Memory" and Name == "AvailableMB"
      | extend TotalMemory = toreal(todynamic(Tags)["vm.azm.ms/memorySizeMB"])
      | extend AvailableMemoryPercentage = (toreal(Val) / TotalMemory) * 100.0
      | summarize AggregatedValue = avg(AvailableMemoryPercentage) by bin(TimeGenerated, 15m), Computer, _ResourceId
    timespan: PT1H
    fields:
      - name: AggregatedValue
        type: value
      - name: Computer
        type: id
    defaultField:
      type: ignore
  # MISSING HEARTBEAT
  - metric: azure_metrics_loganalytics_exporter_missing_heartbeat
    query: |-
      Heartbeat 
      | summarize LastHeartbeat=max(TimeGenerated) by Computer
      | where LastHeartbeat < ago(5h)
      | where Computer !contains_cs "avd"
    fields:
      - name: Computer
        type: id

So the last query is the one I'm struggling with. I have tried specifying the timespan as well, with no luck, in addition to adding multiple fields. Not sure what is required to make this visible.

Just to summarize quickly:

It's working great - but I'm having issues with specific queries and I'm a bit clueless as to why 🙈

[mblaschke]

you have to specify either the timeframe for the query or as part of the query, i'm not sure what azure is doing without timeframe 🤔
if you execute the query in Azure portal check the timeframe and check how many results you're getting from the api.
The exporter relies on the results from the API, without results you don't get metrics.

[deggja]

@mblaschke I see..

So when I'm using this particular query:

 - metric: azure_metrics_loganalytics_exporter_missing_heartbeat
    query: |-
      Heartbeat 
      | summarize LastHeartbeat=max(TimeGenerated) by Computer
      | where LastHeartbeat < ago(5h)
      | where Computer !contains_cs "avd"
    fields:
      - name: Computer
        type: id

The exporter debug flag is giving me the following:

DEBU[0008]/home/runner/work/azure-loganalytics-exporter/azure-loganalytics-exporter/loganalytics/prober.go:358 loganalytics.(*LogAnalyticsProber).executeQueries fetched 4 results                             metric=azure_metrics_loganalytics_exporter_missing_heartbeat module= results=4

So supposedly its fetching four results. If I check log analytics in the Azure Portal it's showing four results, so this is correct. But when I navigate to http://localhost:8080/probe to check the results, the results for that particular azure_metrics_loganalytics_exporter_missing_heartbeat query is not listed.. the others are there as expected.

So I'm getting four results, but no metrics.

[mblaschke]

Could you post the result from LogAnalytics query execution with timeframe "set in query" via Azure portal?

can you try:

 - metric: azure_metrics_loganalytics_exporter_missing_heartbeat
    query: |-
      Heartbeat 
      | summarize LastHeartbeat=max(TimeGenerated) by Computer
      | where LastHeartbeat < ago(5h)
      | where Computer !contains_cs "avd"
    value: 1
    fields:
      - name: Computer
        type: id

[deggja]

Well, it seems the value: 1 solves my issue. Would you care to explain how that is?

Thanks a lot for your super fast and great help @mblaschke - amazing!

[mblaschke]

you cannot publish metrics without values as these wouldn't make sense. but i guess i have to add an error message somewhere so it's obvious.

if you want to use a value from the query you have to define one field as value field

[deggja]

I see. I tried that though, but as there is no obvious value for something like a heartbeat, I used _ResourceId etc. Must the value be integers for it to work? Because using _ResourceId, which I suspect is a string, did not work.

[mblaschke]

Prometheus (and so the prometheus client) is expecting a float64 value so integer is also working fine here.

For "info" metrics you can specify a fixed value for the query (also used eg in azure-resourcegraph-exporter if you want to export resource information).

[deggja]

I see!

I've learned something new today as well. Again, thanks for helping out 🚀