Metrics Opentelemetry sink is failing to push metrics with Bad Request error

[st-omarkhalid]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Pushing to otel-collector is failing with Bad Request

2025-07-11T20:49:05.925571Z DEBUG sink{component_kind="sink" component_id=otel_sink component_type=opentelemetry}:request{request_id=535}:http: vector::internal_events::http_client: HTTP response. status=400 Bad Request version=HTTP/1.1 headers={"content-type": "application/json", "date": "Fri, 11 Jul 2025 20:49:05 GMT", "content-length": "192"} body=[192 bytes]

Known issue? Any work around for this?

Configuration

data_dir: /var/lib/vector
expire_metrics_secs: 600
api:
  enabled: true
  address: 127.0.0.1:8686
  playground: false
sources:
  host_metrics_source:
    type: host_metrics
    scrape_interval_secs: 60
  internal_metrics_source:
    type: internal_metrics
    scrape_interval_secs: 60
  vector_source:
    type: vector
    address: 0.0.0.0:9556
sinks:
  otel_sink:
    type: opentelemetry
    inputs:
    - otel_metrics_filter
    buffer:
      type: memory
      when_full: drop_newest
    protocol:
      type: http
      uri: http://otel-collector.monitoring.svc.cluster.local:4318/v1/metrics
      encoding:
        codec: json
      compression: gzip
      auth:
        strategy: basic
        user: <user>
        password: <password>
  prometheus_exporter:
    type: prometheus_exporter
    inputs:
    - host_metrics_source
    - internal_metrics_source
    address: 0.0.0.0:9598
transforms:
  otel_metrics_filter:
    type: filter
    inputs:
    - vector_source
    condition: .name == .name

Version

v0.49.0

Debug Output

2025-07-11T20:49:05.925571Z DEBUG sink{component_kind="sink" component_id=otel_sink component_type=opentelemetry}:request{request_id=535}:http: vector::internal_events::http_client: HTTP response. status=400 Bad Request version=HTTP/1.1 headers={"content-type": "application/json", "date": "Fri, 11 Jul 2025 20:49:05 GMT", "content-length": "192"} body=[192 bytes]

Example Data

No response

Additional Context

No response

References

No response

[st-omarkhalid]

I tried protobuf as well with no success - there's no Bad Request error but I don't see anything on the otel side

data_dir: /var/lib/vector
expire_metrics_secs: 600
log_schema:
  host_key: "host"
  message_key: "message"
  timestamp_key: "timestamp"

sources:
  internal_metrics:
    type: "internal_metrics"
  
  # Optional: Add other sources like logs or host metrics
  host_metrics:
    type: "host_metrics"
    collectors:
      - cpu
      - memory
      - disk
      - network

transforms:
  # Filter for only specific metrics
  filter_metrics:
    type: "filter"
    inputs: ["internal_metrics", "host_metrics"]
    condition: .name == .name

  # Add labels
  add_labels:
    type: "remap"
    inputs: ["filter_metrics"]
    source: |
      .tags.environment = "test"
      .tags.service = "vector"

sinks:  
  otel_collector:
    inputs: ["add_labels"]
    type: opentelemetry
    protocol:
      type: http
      uri: "http://otel-collector.monitoring.svc.cluster.local:4318/v1/metrics"
      method: post
      framing:
        method: bytes
      encoding:
        codec: protobuf
        protobuf:
          desc_file: /etc/vector/metrics_descriptor.pb
          message_type: opentelemetry.proto.metrics.v1.MetricsData
      headers:
        content-type: application/x-protobuf

Logs

025-07-12T04:45:18.782226Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Couldn't send utilization start wait message. component_id=ComponentKey { id: "otel_collector" } error="Closed(..)"
2025-07-12T04:45:18.782275Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Internal log [Couldn't send utilization start wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:19.782205Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] has been suppressed 10 times.
2025-07-12T04:45:19.782237Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Couldn't send utilization stop wait message. component_id=ComponentKey { id: "filter_metrics" } error="Closed(..)"
2025-07-12T04:45:19.782454Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization start wait message.] has been suppressed 10 times.
2025-07-12T04:45:19.782462Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Couldn't send utilization start wait message. component_id=ComponentKey { id: "filter_metrics" } error="Closed(..)"
2025-07-12T04:45:19.782484Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] has been suppressed 10 times.
2025-07-12T04:45:19.782486Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Couldn't send utilization stop wait message. component_id=ComponentKey { id: "add_labels" } error="Closed(..)"
2025-07-12T04:45:19.782589Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization start wait message.] has been suppressed 10 times.
2025-07-12T04:45:19.782595Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Couldn't send utilization start wait message. component_id=ComponentKey { id: "add_labels" } error="Closed(..)"
2025-07-12T04:45:19.782636Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] has been suppressed 10 times.
2025-07-12T04:45:19.782643Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Couldn't send utilization stop wait message. component_id=ComponentKey { id: "otel_collector" } error="Closed(..)"
2025-07-12T04:45:20.781544Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:20.781763Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization start wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:20.781819Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:20.781990Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization start wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:20.782144Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:29.782435Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Internal log [Couldn't send utilization start wait message.] has been suppressed 25 times.
2025-07-12T04:45:29.782462Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Couldn't send utilization start wait message. component_id=ComponentKey { id: "otel_collector" } error="Closed(..)"
2025-07-12T04:45:29.782552Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Internal log [Couldn't send utilization start wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:30.781301Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] has been suppressed 11 times.
2025-07-12T04:45:30.781337Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Couldn't send utilization stop wait message. component_id=ComponentKey { id: "filter_metrics" } error="Closed(..)"
2025-07-12T04:45:30.781583Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization start wait message.] has been suppressed 11 times.
2025-07-12T04:45:30.781599Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Couldn't send utilization start wait message. component_id=ComponentKey { id: "filter_metrics" } error="Closed(..)"
2025-07-12T04:45:30.781648Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] has been suppressed 11 times.
2025-07-12T04:45:30.781657Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Couldn't send utilization stop wait message. component_id=ComponentKey { id: "add_labels" } error="Closed(..)"
2025-07-12T04:45:30.781863Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization start wait message.] has been suppressed 11 times.
2025-07-12T04:45:30.781871Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Couldn't send utilization start wait message. component_id=ComponentKey { id: "add_labels" } error="Closed(..)"
2025-07-12T04:45:30.781917Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] has been suppressed 11 times.
2025-07-12T04:45:30.781925Z DEBUG sink{component_kind="sink" component_id=otel_collector component_type=opentelemetry}: vector::utilization: Couldn't send utilization stop wait message. component_id=ComponentKey { id: "otel_collector" } error="Closed(..)"
2025-07-12T04:45:31.781724Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:31.781857Z DEBUG transform{component_kind="transform" component_id=filter_metrics component_type=filter}: vector::utilization: Internal log [Couldn't send utilization start wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:31.781882Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization stop wait message.] is being suppressed to avoid flooding.
2025-07-12T04:45:31.781984Z DEBUG transform{component_kind="transform" component_id=add_labels component_type=remap}: vector::utilization: Internal log [Couldn't send utilization start wait message.] is being suppressed to avoid flooding.
2```