v4.1.0

A security and reliability release. It closes several ways an authenticated agent could interfere with another agent on the same proxy, and fixes agent failover for a proxy that accepts connections but rejects registration. It also bounds what clients and agents can ask of the proxy and hardens the dashboard, which adds a few settings. The wire protocol gains two optional fields and a new enum, which older agents and proxies ignore. Some defaults, metric labels, and startup checks change, so read Before you upgrade first.

Highlights

One agent can no longer answer another agent's scrapes. The proxy matched an incoming scrape result to its waiting request by scrape ID alone, and scrape IDs come from one counter shared by every agent, so an agent holding a valid token could answer, fail, or break the chunked transfer of a scrape sent to someone else. Results are now accepted only from the agent the scrape went to, and heartbeats are bound to their connection.

Deployments behind a reverse proxy get the same protection, as far as their identities allow. With transportFilterDisabled, calls are now bound to the auth identity the agent connected with. Agents sharing one identity — including every agent on the legacy proxy.agentToken — still can't be told apart, so give each agent its own proxy.auth token.

A rejected path no longer takes the whole agent offline, and a proxy that rejects an agent is now failed over. One unauthorized path used to disconnect the agent, which then reconnected forever with every path down; now the path is skipped and the rest register, and a rejection that can clear is retried on a backoff. A proxy that accepted the connection but refused registration kept the agent pinned to it; the agent now moves to the next endpoint, and fails back to the primary only after a registered connection drops.

The proxy bounds what clients and agents can ask of it. Each agent's scrape backlog is capped, as are scrapes in flight across all agents, and one agent connection may register at most 20,000 paths (proxy.internal.maxPathsPerAgent), with limits on path length and label size. Another identity can no longer take over or join a path whose agent is still live.

Before you upgrade

• gRPC reflection is off by default. A grpcurl or other reflection client needs proxy.reflectionDisabled = false, and a token when the proxy requires one.
• Scrape outcome labels changed. A scrape the proxy fails itself is no longer upstream_error (502): it is agent_disconnected or proxy_stopped (503), or the new invalid_response (502); client_cancelled (499) is new too. Update alerts and dashboards that count upstream_error.
• Redirects to another origin are no longer followed. Point a target's url at its final location.
• Consolidated agents under different proxy.auth identities can no longer share a path. Put them on one identity.
• Some settings that were accepted now fail at startup, such as a scrapeRequestBacklogUnhealthySize of 0 or a chunk size above gRPC's 4 MiB message limit.
• TLS now uses BoringSSL on the common platforms, so defaults such as cipher suites follow it, and each fat JAR is about 8 MB larger (41 MB → 49 MB).
• Embedders: grpc-netty-shaded is no longer brought in, Agent.startAsyncAgent now waits for startup and throws if it fails, and a redacted ConfigLoadException's cause is a stand-in exception rather than Typesafe's ConfigException.
• Native histograms aren't available through the proxy. The agent no longer asks targets for the protobuf format, which the proxy can't carry.

Also in this release

• More hardening: credentials in target and config URLs are redacted everywhere they are logged, sent, or shown; service discovery drops __-prefixed agent labels (and optionally job / instance); the scrape port serves only exposition content types with nosniff and a sandbox CSP; the dashboard can refuse unknown Host names (proxy.dashboard.allowedHosts); and a connection that hasn't authenticated no longer appears as a connected agent.
• The agent stops a scrape at Prometheus's own timeout, forwarded from X-Prometheus-Scrape-Timeout-Seconds.
• 28 bug fixes, among them corrupted protobuf scrapes, discovered paths that got stuck, a retired path's metrics lingering on /metrics, an agent staying half-connected after a stream failed, scrapes Prometheus abandoned leaving no trace, an HTTP client leaked when a scrape was cancelled while the client cache was busy, and a just-disconnected agent's scrapes being counted as agent_backlog_full.
• Formal checks of the concurrency, alongside the tests: Lincheck specs for the proxy's scrape hand-off, in-flight tracking, path registration, and per-path metrics and for the agent's backlog count, path registrations, and HTTP client cache (make lincheck-tests), and TLA+ models of the proxy's registry and the agent's failover (make tla-checks). They found the last two bugs above.
• Tests run on Java 25, the Docker images' runtime, while the artifact still targets Java 17; every GitHub Action is pinned to a commit SHA; and the docs and website are updated throughout, including new security documentation.
• Dependency updates: Kotlin 2.4.10 → 2.4.20, gRPC 1.83.1 → 1.84.0, Ktor 3.5.2 → 3.6.0, common-utils 3.2.2 → 4.1.0, protobuf 3.25.3 → 3.25.9, Logback 1.6.1 → 1.6.3, SLF4J 2.0.18 → 2.0.20, and netty-tcnative-boringssl-static 2.0.75 → 2.0.81.

Artifacts

• prometheus-proxy.jar / prometheus-agent.jar — standalone fat JARs, attached below
• Maven Central: com.pambrose:prometheus-proxy:4.1.0
• Docker: pambrose/prometheus-proxy:4.1.0, pambrose/prometheus-agent:4.1.0

Full release notes are in RELEASE_NOTES.md; the itemized list is in CHANGELOG.md.

Full Changelog: 4.0.1...4.1.0