feat(remotewriter): mark remote write POSTs with idempotency key and attempt

[vporoshok]

Why

Replaying a remote write POST is a regular part of the delivery contract here: Iterator.SendMessage
resends undelivered messages of a batch with the very same payload, and the Go transport may replay a
request on its own. Nothing on the wire said so:

• a receiver could not tell a replay from new data;
• Retry-Attempt was hardcoded to 0 in protobufWriter.Write, so the attempt number never left the
  process, even though remote.Client.Store already knows how to send it;
• the failed send log was a bare failed to send protobuf: <err> — no endpoint, no attempt, no size.

What

Delivery marks. Both marks of a single delivery travel in the request context (delivery.go) and
reach the wire as headers:

• X-Idempotency-Key = <destination>/<headID>/<targetSegmentID>/<messageIndex>, set by
  deliveryRoundTripper. Messages of a batch are numbered, and a non-empty batch always advances
  targetSegmentID, so the key repeats over all the retries of that message.
• Retry-Attempt, passed to Store as the attempt number of the send loop (0 for the first delivery).

Upstream files are untouched: the round tripper is wrapped around the transport of the client built in
createClient.

Semantics worth knowing. This exact header name also opts the POST into replays made by the Go
transport itself — Request.isReplayable
treats a POST carrying Idempotency-Key or X-Idempotency-Key as replayable, so http.Transport
resends it when a reused connection fails (stale keep-alive, server closing an idle connection, HTTP/2
refused stream). Those replays happen inside a single Client.Do, below our round tripper, and go out
byte for byte identical — same key, same Retry-Attempt. Hence:

• the idempotency key is the only unit of deduplication, the pair of key and attempt is not unique;
• Retry-Attempt is a diagnostic of our send loop, not a count of times a payload reached the receiver.

This is documented in the IdempotencyKeyHeader doc comment and in the package README.

New metric. prometheus_remote_storage_connections_total{remote_name,url,state="new"|"reused"}
counts the connections requests go through (httptrace.GotConn). Since the trace fires per transport
attempt, it also exposes the replays above: more than one connection per send means the transport
resent the request. Growth of state="new" under steady traffic means connections do not survive
keep-alive — the usual source of silent duplicates.

Send error log now reports everything needed to locate the message:

failed to send message: server returned HTTP status 503 Service Unavailable: upstream busy; \
url=http://remote.test/api/v1/write idempotency_key=dst/head-42/7/0 attempt=0 \
duration=1.204s bytes=524288 samples=2000

The URL is the redacted destination URL, the same value as in the metric labels.

Limitations

The key is stable within the life of an iterator. After a restart the same segments may be cut into
different batches (targetSegmentID and the number of shards float), so keys of the re-read data may
differ from those of the first run. A content-derived key would fix that and can be a follow-up.

No receiver reads either header yet.

Tests

delivery_test.go, all green under go test -tags stringlabels -race:

• a replayed message carries the key of its first attempt and the next attempt number
  ({dst/head-42/7/0, 0}, {dst/head-42/7/1, 0}, then {dst/head-42/7/0, 1}), and the failed attempt
  is logged with url / attempt / duration / bytes / samples;
• protobufWriter passes the attempt from the context to Store, and 0 when there are no marks;
• end to end through createClient + httptest: both headers arrive on the wire (this also proves the
  wrapper is not lost behind otelhttp);
• connection counting: the first send reports state="new", the second state="reused".

🤖 Generated with Claude Code