Skip to content

v4.2.0

Choose a tag to compare

@pambrose pambrose released this 26 Sep 19:22
1c0a737

A release that moves the proxy and agent to the Prometheus Java client 1.x, replacing the unmaintained 0.x client, and adds new ways to install and monitor them. Every proxy_* and agent_* metric keeps its name, labels and buckets, but the _created series are gone and, with the JVM exports on, the JVM memory metrics are renamed, so read Before you upgrade first.

Highlights

The Prometheus client underneath moves to 1.x. The proxy and agent now use the Prometheus Java client 1.9.0 (prometheus-metrics-core) through common-utils 5.0.0, instead of simpleclient 0.16.0, which gets no further releases. The /metrics endpoints can also answer protobuf when a scraper asks for it; the histograms stay classic-only.

Install with Homebrew or Helm. brew install pambrose/tap/prometheus-proxy and brew install pambrose/tap/prometheus-agent install commands that run the release JARs on openjdk@25, with brew services definitions to run them in the background. Helm charts for both are in charts/: non-root with a read-only root filesystem, admin-endpoint probes, and an optional LoadBalancer for agents in other clusters, Prometheus Operator ServiceMonitor, and agent token from a Secret.

The Grafana dashboards and alerts work again, and ship as files. The dashboards and the alert rules queried seven counters by their bare names, but a counter is exposed only as <name>_total, so the connection, eviction, heartbeat and scrape-count panels showed no data. They now use the _total names, and a test checks every query against the series the proxy and agent expose. The alerting rules ship as grafana/alerts.yml, checked with promtool in CI, and add ProxyDown. The proxy dashboard gains Job and Instance pickers, so two proxies no longer add up in every panel, and the agents dashboard tells agents apart by job and gains an Unsuccessful Scrapes by Agent panel.

Security. The Netty that gRPC brought (4.2.16, with a critical CVE) is raised to 4.2.18, and the Jackson that the Dropwizard metrics servlets brought (2.12.7, with high CVEs) to 2.22.3. A new security workflow scans both images with Trivy on every pull request and weekly, and CodeQL analyzes the code. SECURITY.md, linked from the website's Security page, explains how to report a vulnerability privately through GitHub.

Before you upgrade

  • The _created series of every counter and histogram are gone. Set IO_PROMETHEUS_EXPORTER_INCLUDE_CREATED_TIMESTAMPS=true to bring them back.
  • With the JVM exports on, the memory metrics put the unit last: jvm_memory_bytes_used is now jvm_memory_used_bytes, jvm_memory_pool_bytes_used is jvm_memory_pool_used_bytes, and the same for committed, max and init. jvm_info is jvm_runtime_info. Update dashboards and alerts that use them.
  • Re-import the Grafana dashboards, and scrape each agent's own metrics in its own job, through the proxy: the agents dashboard tells agents apart by job.
  • Embedders: the agent brings io.prometheus:prometheus-metrics-core 1.9.0 instead of simpleclient, and registers its metrics in the 1.x default registry, so a host that serves the 0.x registry no longer exposes them. prometheus-metrics-simpleclient-bridge lets a host serve both from the 1.x registry; the embedded-agent page's Upgrading from 4.1.x section has the details.
  • The proxy.metrics.grpc and agent.metrics.grpc settings are gone. Nothing ever read them; a config that still sets them keeps loading.

Also in this release

  • Quieter startup: the JARs no longer print logback's internal status or kotlin-logging's initialization line, and the proxy no longer warns that its stale-agent cleanup "was added after Proxy was initialized"; the cleanup service is now covered by the all_services_healthy health check.
  • -u prints the usage in a terminal on Java 17, and -u and -v print only the usage or the version, without the startup banner.
  • Docs: every way of running the proxy and agent says how to run it in the background, the Docker page's production examples run, the Quick Start's Build from Source tab builds the JARs, and the embedded-agent page has an Upgrading from 4.1.x section.
  • CI model-checks the TLA+ specs and runs the container tests on every pull request, installs the Helm charts into a kind cluster, and has a manual Lincheck workflow.
  • Dependency updates: common-utils 4.1.0 → 5.0.0, the Prometheus Java client simpleclient 0.16.0 → prometheus-metrics-core 1.9.0, Netty 4.2.16 → 4.2.18, Jackson 2.12.7 → 2.22.3, Logback 1.6.3 → 1.6.4, and the dashboard's htmx 2.0.10 → 2.0.11; Gradle 9.7.1 → 9.8.0 for the build.

Artifacts

  • prometheus-proxy.jar / prometheus-agent.jar — standalone fat JARs, and prometheus-proxy-4.2.0-sources.jar, attached below
  • Maven Central: com.pambrose:prometheus-proxy:4.2.0
  • Docker: pambrose/prometheus-proxy:4.2.0, pambrose/prometheus-agent:4.2.0
  • Homebrew: pambrose/tap/prometheus-proxy, pambrose/tap/prometheus-agent
  • Helm: charts/prometheus-proxy, charts/prometheus-agent

Full release notes are in RELEASE_NOTES.md; the itemized list is in CHANGELOG.md.

Full Changelog: 4.1.0...4.2.0