BadRequest EventHub Metrics

[IMAMBAKS]

Currently I'm retrieving two metrices:

• Incoming messages
• EventHub size.

The Incoming Messages do work however I'm getting BadRequests for other Metrices e.g. EventHub Size.

Using the following configuration:

metrics:
  - name: azure_event_hubs_incoming_messages
    description: "The number of incoming messages on an Azure Event Hubs topic"
    resourceType: EventHubs
    azureMetricConfiguration:
      metricName: IncomingMessages
      aggregation:
        type: Total
    resources:
      - namespace: xxx-dl-evh-acc

  - name: azure_event_hubs_size
    description: "Size of an EventHub in Bytes."
    resourceType: EventHubs
    azureMetricConfiguration:
      metricName: Size
      aggregation:
        type: Average
    resources:
      - namespace: xxx-dl-evh-acc

metricSinks:
  prometheusScrapingEndpoint:
    enabled: true
    baseUriPath: /metrics
    enableMetricTimestamps: true
    metricUnavailableValue: NaN
    labelTransformation: None
    enableServiceDiscovery: true
    serviceMonitor:
      enabled: true
      namespace: "datalake-monitoring"
      labels: { }
      interval: 30s
      timeout: 10s

metricDefaults:
  aggregation:
    interval: 00:01:00
  scraping:
    schedule: "* * * * *"

Bad request message:
BadRequest: Failed to serialize response for request: https://westeurope.mdmrp.prod.windows.azure.net/subscriptions/xxxxb/resourceGroups/eventhub-acc-rg/providers/Microsoft.EventHub/namespaces/xxx-dl-evh-acc/providers/microsoft.insights/metrics?$filter=EntityName eq '*'&timespan=2022-04-28T07%3A39%3A00.9960246Z%2F2022-05-03T07%3A39%3A00.9960246Z&interval=PT1M&metricnames=Size&aggregation=Average&top=10000&api-version=2018-01-01, Duration: 00:00:00.7031398, TraceId: {4f634207-2197-4058-92c0-ca5932a17b9f}

Error message:

[07:40:02 FTL] Failed to scrape resource for metric 'azure_event_hubs_size'
Microsoft.Azure.Management.Monitor.Fluent.Models.ErrorResponseException: Operation returned an invalid status code 'BadRequest'
   at Microsoft.Azure.Management.Monitor.Fluent.MetricsOperations.ListWithHttpMessagesAsync(String resourceUri, ODataQuery`1 odataQuery, String timespan, Nullable`1 interval, String metricnames, String aggregation, Nullable`1 top, String orderby, Nullable`1 resultType, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricsOperationsExtensions.ListAsync(IMetricsOperations operations, String resourceUri, ODataQuery`1 odataQuery, String timespan, Nullable`1 interval, String metricnames, String aggregation, Nullable`1 top, String orderby, Nullable`1 resultType, String metricnamespace, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.Models.MetricDefinitionImpl.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.Models.MetricDefinitionImpl.Microsoft.Azure.Management.Monitor.Fluent.IWithMetricsQueryExecute.ExecuteAsync(CancellationToken cancellationToken)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.GetRelevantMetric(String metricName, AggregationType metricAggregation, TimeSpan metricInterval, String metricFilter, String metricDimension, IMetricDefinition metricDefinition, Nullable`1 metricLimit, DateTime recordDateTime) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 183
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter, Nullable`1 metricLimit) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 90
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 72
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 103

[tomkerkhove]

Odd, I will have to check that.

Would you mind enabling this to see if you get more feedback?

https://docs.promitor.io/v2.6/scraping/runtime-configuration/?h=monitor#azure-monitor

[IMAMBAKS]

I added the following to my config:

azureMonitor:
  logging:
    informationLevel: Basic # Optional. Default: Basic
    isEnabled: true # Optional. Default: false

I don't see "more" information in the logs.
I'm getting the same error message as can be seen here below. You can also see some successfull retrieval of incoming messages..

[09:16:02 FTL] BadRequest: Failed to serialize response for request: https://westeurope.mdmrp.prod.windows.azure.net/subscriptions/xxx/resourceGroups/eventhub-acc-rg/providers/Microsoft.EventHub/namespaces/xxx-dl-evh-acc/providers/microsoft.insights/metrics?$filter=EntityName eq '*'&timespan=2022-04-30T09%3A16%3A00.4876959Z%2F2022-05-05T09%3A16%3A00.4876959Z&interval=PT1M&metricnames=Size&aggregation=Average&top=10000&api-version=2018-01-01, Duration: 00:00:00.8906296, TraceId: {f64d3761-b289-4ff3-8e61-a186e8dde99f}
[09:16:03 INF] Found value 2132 for metric azure_event_hubs_incoming_messages with dimension xxx as part of entity_name dimension with aggregation interval 00:01:00
[09:16:03 INF] Found value 0 for metric azure_event_hubs_incoming_messages with dimensionxxx as part of entity_name dimension with aggregation interval 00:01:00
[09:16:03 INF] Found value 0 for metric azure_event_hubs_incoming_messages with dimension xxx as part of entity_name dimension with aggregation interval 00:01:00
[09:16:03 INF] Found value 0 for metric azure_event_hubs_incoming_messages with dimension xxx as part of entity_name dimension with aggregation interval 00:01:00
[09:16:03 INF] Found value 0 for metric azure_event_hubs_incoming_messages with dimension xxx as part of entity_name dimension with aggregation interval 00:01:00
[09:16:03 INF] Found value 0 for metric azure_event_hubs_incoming_messages with dimensionxxx as part of entity_name dimension with aggregation interval 00:01:00
[09:16:03 INF] Found value 0 for metric azure_event_hubs_incoming_messages with dimension xxx as part of entity_name dimension with aggregation interval 00:01:00

However .... sometimes I'm also getting a BadRequest for the incoming messages as can be seen here below:

 Failed to scrape resource for metric 'azure_event_hubs_incoming_messages'
Microsoft.Azure.Management.Monitor.Fluent.Models.ErrorResponseException: Operation returned an invalid status code 'BadRequest'
   at Microsoft.Azure.Management.Monitor.Fluent.MetricsOperations.ListWithHttpMessagesAsync(String resourceUri, ODataQuery`1 odataQuery, String timespan, Nullable`1 interval, String metricnames, String aggregation, Nullable`1 top, String orderby, Nullable`1 resultType, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricsOperationsExtensions.ListAsync(IMetricsOperations operations, String resourceUri, ODataQuery`1 odataQuery, String timespan, Nullable`1 interval, String metricnames, String aggregation, Nullable`1 top, String orderby, Nullable`1 resultType, String metricnamespace, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.Models.MetricDefinitionImpl.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.Models.MetricDefinitionImpl.Microsoft.Azure.Management.Monitor.Fluent.IWithMetricsQueryExecute.ExecuteAsync(CancellationToken cancellationToken)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.GetRelevantMetric(String metricName, AggregationType metricAggregation, TimeSpan metricInterval, String metricFilter, String metricDimension, IMetricDefinition metricDefinition, Nullable`1 metricLimit, DateTime recordDateTime) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 183
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter, Nullable`1 metricLimit) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 90
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 72
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 103

[tomkerkhove]

Hm sorry to hear - I'l ltry to dive in to this asap.

[dks0296586]

I am also receiving BadRequest responses for EventHub, but they are slightly different errors

   - name: azure_eventhubs_activeconnections_average
      description: "Total Active Connections for Microsoft.EventHub."
      resourceType: EventHubs
      azureMetricConfiguration:
        metricName: ActiveConnections
        aggregation:
          type: Average
      resourceDiscoveryGroups:
      - name: event-hubs-landscape 
      
   - name: azure_eventhubs_activeconnections_maximum
      description: "Total Active Connections for Microsoft.EventHub."
      resourceType: EventHubs
      azureMetricConfiguration:
        metricName: ActiveConnections
        aggregation:
          type: Maximum
      resourceDiscoveryGroups:
      - name: event-hubs-landscape

[14:48:27 FTL] BadRequest: Metric: ActiveConnections does not support requested dimension combination: entityname, supported ones are: , TraceId: {xxxxx}
[14:48:29 FTL] BadRequest: Metric: ActiveConnections does not support requested dimension combination: entityname, supported ones are: , TraceId: {xxxxx}
[14:48:30 FTL] BadRequest: Metric: ActiveConnections does not support requested dimension combination: entityname, supported ones are: , TraceId: {xxxxx}

I noticed that the log message is indicating that the metric provided the dimension, entityname, however in my metric definition i do not. Also, microsoft docs(and log message) indicates there is no dimension supported for this metric.

[tomkerkhove]

Ha, but you are using resource discovery which will use that dimension under the hood I think. Let me check this as it might be a bug/known limitation.

[IMAMBAKS]

hi @tomkerkhove. Since this week we also have this problem with the incoming messages.
We have found THE problem but we cannot solve it on our side.

When we are looking at the URL, the timespan covers 5 days! but the interval is 1 minute. We tested it with a smaller interval (1 minute). this worked fine for us. See image below:

[tomkerkhove]

Interesting, seems like the requests are very greedy (by SDK or me). I'll check this ASAP

[tomkerkhove]

I started working on a fix: #2103