Prometheus Remote‑Write Metadata Handling

[nickvanw]

I wasn't sure if I should file this as a bug or a feature - there's aspects of both, but I wanted to get this in front of everyone before going in any particular direction.

What I’m seeing

While forwarding Prometheus metrics through Vector’s prometheus_remote_write source, every metric that doesn’t have metadata in the same request is ingested as Untyped.
Counters, gauges, histograms, etc. all lose their original type.

Why this is surprising

1. Docs say otherwise
   The current docs for prometheus_remote_write state that:

   “the remote_write protocol only transmits the metric’s tags, timestamp, and value … Vector guesses the metric type (e.g. names ending in _total are counters).”

   After reading the source code (prometheus-parser/src/lib.rs), that heuristic isn’t implemented. The parser:

   • records any MetricMetadata only if it’s in the same WriteRequest as the samples;
   • otherwise defaults the metric to MetricKind::Untyped.
     (There’s no suffix‑based promotion to Counter/Gauge.)

Key lines: prometheus-parser/src/lib.rs::parse_request creates a fresh MetricGroupSet, records any MetricMetadata in that request, and otherwise sets MetricKind::Untyped. No name‑based re‑classification occurs.

2. Prometheus never co‑locates metadata
   With remote‑write v1, Prometheus sends metadata (type/help) in separate periodic requests. Samples and metadata are never in the same payload, so Vector always sees metrics without type info → they become Untyped.

3. Downstream impact
   Sinks like Datadog then treat counters/histograms as gauges, which breaks monotonic‑count logic and histogram aggregation.

What I think is happening

Step	Actor	Behaviour
1	Prometheus	Sends samples in one request; sends MetricMetadata in a different request a minute later.
2	Vector	Parses each request independently. If a request lacks metadata, all metrics default to Untyped.
3	Sinks	Receive everything as gauges/untyped.

Suggestions

• Clarify that Vector only applies metric type if the metadata is in the same request, and that no name‑suffix heuristic is currently applied.
• Cache metadata across requests (with a TTL) or adopt Prometheus remote‑write v2 (which aims to make metadata persistent via the WAL or co‑located with samples).
• Vector could do something like New Relic (https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-remote-write/set-your-prometheus-remote-write-integration/#mapping) and allow you to specify a static mapping, either as a label on the metric (which could be rewritten), or via configuration?
• Vector could also guess using OpenMetrics suffixes, as New Relic does. The documentation discusses this behavior, but to my knowledge it is not currently implemented.

Related issues

• All metrics sent to the vector via remote write are of the gauge type #19973 “All metrics sent to Vector via remote write are of the gauge type”
• Monotonic Prometheus counter sent to Datadog as gauge #10171 “Monotonic Prometheus counter sent to Datadog as gauge”
• Prometheus remote write source has incomplete support/behavior. #10801 "Prometheus remote write source has incomplete support/behavior."

Other Stuff

In the course of testing this, I whipped up a basic proof of concept of a middleware service that would annotate requests from a static list of metrics, to make sure that things worked as expected: https://github.com/nickvanw/metrics-typer

I can confirm that when a remote write request comes in with both metadata and data, Vector does the appropriate things and passes types all the way through the system, to the point I was able to get Histograms working in Datadog no problem.