Disabling enableMetricTimestamps should disable timestamps for metadata metrics

[nikosmeds]

Report

Setting prometheusScrapingEndpoint.enableMetricTimestamps to false should disable timestamps for all Prometheus metrics.

Currently the timestamps are disabled for "value" metrics but not "metadata" metrics.

An example of this can be seen below (most labels have been redacted from the example):

$ curl -s promitor-agent-scraper:8888/metrics | grep azure_storage_account_egress_bytes
# HELP azure_storage_account_egress_bytes The amount of egress data, in bytes.
# TYPE azure_storage_account_egress_bytes gauge
azure_storage_account_egress_bytes{resource_type="StorageAccount", ...} 942
promitor_scrape_error{metric_name="azure_storage_account_egress_bytes", ...} 0 1642198339570
promitor_scrape_success{metric_name="azure_storage_account_egress_bytes", ...} 1 1642198339569

Note the appended timestamps on the last two metrics.

Expected Behavior

Metric timestamps would not be appended anywhere.

Actual Behavior

Metric timestamps are appended to the "metadata" metrics.

Steps to Reproduce the Problem

1. Set prometheusScrapingEndpoint.enableMetricTimestamps to false
2. Scrape any Azure Monitor metrics
3. HTTP GET the exporter endpoint to review raw Prometheus metric results

Component

Scraper

Version

2.6.0

Configuration

Configuration:

metrics-declaration.yaml:
----
version: v1
azureMetadata:
  tenantId: REDACTED
  subscriptionId: REDACTED
  resourceGroupName: REDACTED
  cloud: Global
metricDefaults:
  aggregation:
    interval: 00:02:00
  scraping:
    schedule: "*/2 * * * *"
metrics:
- azureMetricConfiguration:
    aggregation:
      type: Total
    metricName: ByteCount
  description: Total number of Bytes transmitted within time period.
  name: azure_load_balancer_traffic_bytes
  resourceDiscoveryGroups:
  - name: load-balancer-landscape
  resourceType: LoadBalancer
- azureMetricConfiguration:
    aggregation:
      type: Total
    metricName: PacketCount
  description: Total number of Packets transmitted within time period.
  name: azure_load_balancer_traffic_packets
  resourceDiscoveryGroups:
  - name: load-balancer-landscape
  resourceType: LoadBalancer
- azureMetricConfiguration:
    aggregation:
      type: Average
    metricName: Availability
  description: The percentage of availability for the storage service or the specified
    API operation.
  name: azure_storage_account_availability_percent
  resourceDiscoveryGroups:
  - name: storage-account-landscape
  resourceType: StorageAccount
- azureMetricConfiguration:
    aggregation:
      type: Total
    metricName: Egress
  description: The amount of egress data, in bytes.
  name: azure_storage_account_egress_bytes
  resourceDiscoveryGroups:
  - name: storage-account-landscape
  resourceType: StorageAccount
- azureMetricConfiguration:
    aggregation:
      type: Total
    metricName: Ingress
  description: The amount of ingress data, in bytes.
  name: azure_storage_account_ingress_bytes
  resourceDiscoveryGroups:
  - name: storage-account-landscape
  resourceType: StorageAccount
- azureMetricConfiguration:
    aggregation:
      type: Average
    metricName: SuccessE2ELatency
  description: The end-to-end latency of successful requests made to a storage service
    or the specified API operation, in milliseconds.
  name: azure_storage_account_success_e2e_latency_milliseconds
  resourceDiscoveryGroups:
  - name: storage-account-landscape
  resourceType: StorageAccount
- azureMetricConfiguration:
    aggregation:
      type: Average
    metricName: SuccessServerLatency
  description: The latency used by Azure Storage to process a successful request,
    in milliseconds.
  name: azure_storage_account_success_server_latency_milliseconds
  resourceDiscoveryGroups:
  - name: storage-account-landscape
  resourceType: StorageAccount
- azureMetricConfiguration:
    aggregation:
      type: Average
    metricName: UsedCapacity
  description: The amount of storage used by the storage account.
  name: azure_storage_account_used_capacity_bytes
  resourceDiscoveryGroups:
  - name: storage-account-landscape
  resourceType: StorageAccount
- azureMetricConfiguration:
    aggregation:
      type: Total
    metricName: Transactions
  description: The number of requests made to a storage service or the specified API
    operation.
  name: azure_storage_account_transactions_count
  resourceDiscoveryGroups:
  - name: storage-account-landscape
  resourceType: StorageAccount

runtime.yaml:
----
server:
  httpPort: "5000"
authentication:
  mode: ServicePrincipal
  identityId: REDACTED
resourceDiscovery:
  host: "promitor-agent-resource-discovery"
  port: 8889
metricSinks:
  prometheusScrapingEndpoint:
    metricUnavailableValue: "NaN"
    enableMetricTimestamps: "false"
    baseUriPath: "/metrics"
    labels:
      transformation: "None"
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
telemetry:
  applicationInsights:
    isEnabled: "false"
  containerLogs:
    isEnabled: "true"
  defaultVerbosity: "Error"

Logs

No response

Platform

Microsoft Azure

Contact Details

No response

[tomkerkhove]

Valid remark, thanks for reporting! Are you willing to contribute this?

[nikosmeds]

Thanks @tomkerkhove. I've added this issue to my "todos", but can't make any promises when I'll get around to it.

Right now we workaround this issue by updating Prometheus' scrape_config with honor_timestamps: false.

honor_timestamps controls whether Prometheus respects the timestamps present in scraped data.

[tomkerkhove]

Sounds good, thanks! Let me know if you need help.