Filter or Transform NaN values

[PKaelin]

Problem

I have a Prometheus receiver in OpenTelemetry Collector that receives Gauge metrics with value NaN (Not a Number) which I try to filter/transform before sending it to an InfluxDB backend.

Note that the example data is at the end of this message.

Receiver

The receiver configuration is below and I believe I cannot change the source (HAProxy) to omit NaN values:

receivers:
 prometheus:
   config:
     scrape_configs:
       - job_name: 'internalgateway'
         scrape_interval: 15s
         static_configs:
           - targets: ['internal_gateway:8405']
         metrics_path: /metrics
         scheme: http

Processors-Filter

I did a test filter configuration in Processor like below but the manual says that conditions that apply to the same signal, such as spans and span events, if the "higher" level telemetry matches a condition and is dropped, the "lower" level condition will not be checked. So in the filter below all the Gauge metrics are dropped.

processors:
  filter/drop_nan_inf:
    error_mode: ignore
    metrics:
      metric:
        - type == METRIC_DATA_TYPE_GAUGE
      datapoint:
       # Not sure how to check for NaN IsDoble allows for NaN
        - datapoint.value_double > -100000000 and datapoint.value_double < 100000000
       # - metric.type == METRIC_DATA_TYPE_GAUGE        # This filter doesn’t work it has to be in the metric section
       # - IsString(datapoint.value_double)
       # - IsMatch(datapoint.value_double, "NaN")
       # - datapoint.value_double > -1.7976931348623157e+308 and datapoint.value_double < 1.7976931348623157e+308

Processors-Transform

I also did a test transform configuration in Processor but that doesn’t seem to set the value even when condition is for all Gauge metric:

processors:
  transform/trans_nan_inf:
    error_mode: ignore
    metric_statements:
      - error_mode: ignore
        conditions:
          - metric.type == METRIC_DATA_TYPE_GAUGE
        statements:
          - set(datapoint.value_double, 123)

So how do I filter/transform metrices with NaN values?

Sample data (Log-Output)

{
    "resource": {
        "service.instance.id": "0945af19-108b-4c69-9d7b-2c3f8d8dc5b0",
        "service.name": "otelcol-contrib",
        "service.version": "0.143.1"
    },
    "otelcol.component.id": "filter/drop_nan_inf",
    "otelcol.component.kind": "processor",
    "otelcol.pipeline.id": "metrics/influxdb",
    "otelcol.signal": "metrics",
    "condition": "metric.type == METRIC_DATA_TYPE_GAUGE",
    "match": true,
    "TransformContext": {
        "resource": {
            "attributes": {
                "service.name": "internalapigateway",
                "server.address": "testdocker.internal_api_gateway",
                "service.instance.id": "testdocker.internal_api_gateway:8405",
                "server.port": "8405",
                "url.scheme": "http"
            },
            "dropped_attribute_count": 0
        },
        "scope": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver",
            "version": "0.143.1"
        },
        "metric": {
            "description": "Limit on the number of connections in queue, for servers only (maxqueue argument)",
            "name": "haproxy_server_queue_limit",
            "unit": "",
            "type": "Gauge",
            "metadata": { "prometheus.type": "gauge" },
            "datapoints": [
                {
                    "attributes": {
                        "proxy": "http_back_operatingmodeservice",
                        "server": "operatingmodeservicesrv"
                    },
                    "exemplars": [],
                    "flags": 0,
                    "start_time_unix_nano": 0,
                    "time_unix_nano": 1767731654829000000,
                    "value_double": "NaN"
                },
                {
                    "attributes": {
                        "proxy": "tcp_back_outstation",
                        "server": "outstationsrv"
                    },
                    "exemplars": [],
                    "flags": 0,
                    "start_time_unix_nano": 0,
                    "time_unix_nano": 1767731654829000000,
                    "value_double": "NaN"
                }
            ]
        },
        "datapoint": {
            "attributes": {
                "proxy": "http_back_operatingmodeservice",
                "server": "operatingmodeservicesrv"
            },
            "exemplars": [],
            "flags": 0,
            "start_time_unix_nano": 0,
            "time_unix_nano": 1767731654829000000,
            "value_double": "NaN"
        },
        "cache": {}
    }
}

[edmocosta]

Hi @PKaelin!

For the Transform processor issue, you need to set it with the right data type value, which in this case is double/float64 (123.0), not integer (123), example:

- set(datapoint.value_double, 123.0)

Unfortunately we don't have a custom function to check whether a value is NaN, but you can do it by comparing
the double value to itself (see IEEE 754). NaNare the only one that satisfy the v != v condition, for example (see):

- datapoint.value_double != datapoint.value_double # filter out NaN values

[PKaelin]

Thanks for the reply and sorry for my late reply. I thought I did try the v !=v condition without any success. Anyway, at the end I found out that HAProxy sends NaN values for fields that I didn't configure in HAProxy as default was ok for me. I ended up filtering those metrics in the OpenTelemetry scrape configuration:

prometheus:
  config:
    scrape_configs:
     - job_name: 'xxxx'
        metric_relabel_configs:
          - source_labels: [__name__]
            regex: "haproxy_xxxxx | haproxy_server_xxxxxx | . . ."
            action: drop

[PKaelin]

Was able to filter the metrics in the scrape configuration instead as there were only a couple of them and only the ones I did not configure in the HAProxy. See solution in the answers.