2.55.0-rc.0 / 2024-09-20

While all the excitement is focused on Prometheus 3.0, we are preparing release 2.55.

This release has quite a few new features, some of which require a feature-flag to enable, a major optimisation of queries when out-of-order data is present, and a number of bug-fixes.

• [FEATURE] Support UTF-8 characters in label names - feature flag utf8-names. #14482, #14880, #14736, #14727
• [FEATURE] Support config reload automatically - feature flag auto-reload-config. #14769
• [FEATURE] Scraping: Add the ability to set custom http_headers in config. #14817
• [FEATURE] Scraping: Support feature flag created-timestamp-zero-ingestion in OpenMetrics. #14356, #14815
• [FEATURE] Scraping: scrape_failure_log_file option to log failures to a file. #14734
• [FEATURE] PromQL: Delay removal of __name__ label - feature flag promql-delayed-name-removal. #14477
• [FEATURE] OTLP receiver: Optional promotion of resource attributes to series labels. #14200
• [FEATURE] API: Support multiple --web.listen-address. #14665
• [FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
• [FEATURE] Promtool: tsdb create-blocks new option to add labels. #14403
• [FEATURE] Promtool: promtool test adds --junit flag to format results. #14506
• [ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
• [ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
• [ENHANCEMENT] PromQL: Improve detail in distributed tracing. #14816
• [ENHANCEMENT] PromQL: Experimental sort_by_label and sort_by_label_desc sort by all labels when label is equal. #14655
• [ENHANCEMENT] PromQL: Clarify error message logged when Go runtime panic occurs during query evaluation. #14621
• [ENHANCEMENT] PromQL: Use Kahan summation for better accuracy in avg and avg_over_time. #14413
• [ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
• [PERF] Remote-Read: Support streaming mode. #11379
• [PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948
• [PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
• [BUGFIX] SD: Fix dropping targets (with feature flag new-service-discovery-manager). #13147
• [BUGFIX] SD: Stop storing stale targets (with feature flag new-service-discovery-manager). #13622
• [BUGFIX] fix(utf8): propagate validationScheme config to scraping options. #14880
• [BUGFIX] PromQL: Experimental Native Histograms: Do not re-use spans between histograms. #14771
• [BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
• [BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
• [BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
• [BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14598, #14611, #14609, #14575, #14513, #14821
• [BUGFIX] TSDB: Only count unknown record types in record_decode_failures_total metric. #14042
• [ENHANCEMENT] Remote-Write: Don't reshard if we haven't successfully sent a sample since last update. #14450

3.0.0-beta.0 / 2024-09-05

Release 3.0.0-beta.0 includes new features such as a brand new UI and UTF-8 support enabled by default. As a new major version, several breaking changes are introduced. The breaking changes are mainly around the removal of deprecated feature flags and CLI arguments, and the full list can be found below. Most users should be able to try this release out of the box without any configuration changes.

As is traditional with a beta release, we do not recommend users install 3.0.0-beta on critical production systems, but we do want everyone to test it out and find bugs.

• [CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch back to the old UI, you can use the feature flag --enable-feature=old-ui for the time being. #14872
• [CHANGE] PromQL: Range selectors and the lookback delta are now left-open, i.e. a sample coinciding with the lower time limit is excluded rather than included. #13904
• [CHANGE] Kubernetes SD: Remove support for discovery.k8s.io/v1beta1 API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365
• [CHANGE] Kubernetes SD: Remove support for networking.k8s.io/v1beta1 API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365
• [CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed. #14705
• [CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807
• [CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed. #14770
• [CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead. #14747
• [CHANGE] Remove deprecated remote-write-receiver,promql-at-modifier, and promql-negative-offset feature flags. #13456, #14526
• [CHANGE] Remove deprecated storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention flags. #14640, #14643
• [FEATURE] Promtool: Allow additional labels to be added to blocks created from openmetrics. #14402
• [FEATURE] OTLP receiver: Add new option otlp.promote_resource_attributes, for any OTel resource attributes that should be promoted to metric labels. #14200
• [FEATURE] Automatic reloading of the Prometheus configuration file at a specified interval #14769
• [ENHANCEMENT] OTLP receiver: Warn when encountering exponential histograms with zero count and non-zero sum. #14706
• [ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
• [ENHANCEMENT] Scrape: Only parse created timestamp if created-timestamp-zero-ingestion feature flag is enabled. This is as a lot of memory is used when parsing the created timestamp in the OM text format. #14815
• [ENHANCEMENT] Scrape: Add support for logging scrape failures to a specified file. #14734
• [ENHANCEMENT] Remote Read client: Enable streaming remote read if the server supports it. #11379
• [ENHANCEMENT] PromQL: Delay deletion of __name__ label to the end of the query evaluation. This is experimental and enabled under the feature-flag promql-delayed-name-removal. #14477
• [ENHANCEMENT] Move AM discovery page from "Monitoring status" to "Server status". #14875
• [ENHANCEMENT] Tracing: Improve PromQL tracing, including showing the operation performed for aggregates, operators, and calls.#14816
• [ENHANCEMENT] Add support for multiple listening addresses. #14665
• [ENHANCEMENT] Add the ability to set custom HTTP headers. #14817
• [BUGFIX] TSDB: Fix shard initialization after WAL repair. #14731
• [BUGFIX] UTF-8: Ensure correct validation when legacy mode turned on. #14736
• [BUGFIX] SD: Make discovery manager notify consumers of dropped targets for still defined jobs. #13147
• [BUGFIX] SD: Prevent the new service discovery manager from storing stale targets. #13622
• [BUGFIX] Remote Write 2.0: Ensure metadata records are sent from the WAL to remote write during WAL replay. #14766
• [BUGFIX] Scrape: Do no override target parameter labels with config params. #11029
• [BUGFIX] Scrape: Reset exemplar position when scraping histograms in protobuf. #14810
• [BUGFIX] Native Histograms: Do not re-use spans between histograms. #14771
• [BUGFIX] Scrape: Only parse created timestamp if created-timestamp-zero-ingestion feature flag is enabled. This is as a lot of memory is used when parsing the created timestamp in the OM text format. #14815
• [BUGFIX] TSDB: Fix panic in query during truncation with OOO head. #14831
• [BUGFIX] TSDB: Fix panic in chunk querier. #14874
• [BUGFIX] promql.Engine.Close: No-op if nil. #14861
• [BUGFIX] tsdb/wlog.Watcher.readSegmentForGC: Only count unknown record types against record_decode_failures_total metric. #14042

2.54.1 / 2024-08-27

• [BUGFIX] Scraping: allow multiple samples on same series, with explicit timestamps. #14685
• [BUGFIX] Docker SD: fix crash in match_first_network mode when container is reconnected to a new network. #14654
• [BUGFIX] PromQL: fix experimental native histogram counter reset detection on stale samples. #14514
• [BUGFIX] PromQL: fix experimental native histograms getting corrupted due to vector selector bug in range queries. #14538
• [BUGFIX] PromQL: fix experimental native histogram memory corruption when using histogram_count or histogram_sum. #14605

Full Changelog: v2.54.0...v2.54.1

2.54.0 / 2024-08-09

Release 2.54 brings a release candidate of a major new version of Remote Write: 2.0.
This is experimental at this time and may still change.
Remote-write v2 is enabled by default, but can be disabled via feature-flag web.remote-write-receiver.accepted-protobuf-messages.

• [CHANGE] Remote-Write: highest_timestamp_in_seconds and queue_highest_sent_timestamp_seconds metrics now initialized to 0. #14437
• [CHANGE] API: Split warnings from info annotations in API response. #14327
• [FEATURE] Remote-Write: Version 2.0 experimental, plus metadata in WAL via feature flag metadata-wal-records (defaults on). #14395,#14427,#14444
• [FEATURE] PromQL: add limitk() and limit_ratio() aggregation operators. #12503
• [ENHANCEMENT] PromQL: Accept underscores in literal numbers, e.g. 1_000_000 for 1 million. #12821
• [ENHANCEMENT] PromQL: float literal numbers and durations are now interchangeable (experimental). Example: time() - my_timestamp > 10m. #9138
• [ENHANCEMENT] PromQL: use Kahan summation for sum(). #14074,#14362
• [ENHANCEMENT] PromQL (experimental native histograms): Optimize histogram_count and histogram_sum functions. #14097
• [ENHANCEMENT] TSDB: Better support for out-of-order experimental native histogram samples. #14438
• [ENHANCEMENT] TSDB: Optimise seek within index. #14393
• [ENHANCEMENT] TSDB: Optimise deletion of stale series. #14307
• [ENHANCEMENT] TSDB: Reduce locking to optimise adding and removing series. #13286,#14286
• [ENHANCEMENT] TSDB: Small optimisation: streamline special handling for out-of-order data. #14396,#14584
• [ENHANCEMENT] Regexps: Optimize patterns with multiple prefixes. #13843,#14368
• [ENHANCEMENT] Regexps: Optimize patterns containing multiple literal strings. #14173
• [ENHANCEMENT] AWS SD: expose Primary IPv6 addresses as __meta_ec2_primary_ipv6_addresses. #14156
• [ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks. #10490
• [ENHANCEMENT] OpenStack SD: Use flavor.original_name if available. #14312
• [ENHANCEMENT] UI (experimental native histograms): more accurate representation. #13680,#14430
• [ENHANCEMENT] Agent: out_of_order_time_window config option now applies to agent. #14094
• [ENHANCEMENT] Notifier: Send any outstanding Alertmanager notifications when shutting down. #14290
• [ENHANCEMENT] Rules: Add label-matcher support to Rules API. #10194
• [ENHANCEMENT] HTTP API: Add url to message logged on error while sending response. #14209
• [BUGFIX] CLI: escape | characters when generating docs. #14420
• [BUGFIX] PromQL (experimental native histograms): Fix some binary operators between native histogram values. #14454
• [BUGFIX] TSDB: LabelNames API could fail during compaction. #14279
• [BUGFIX] TSDB: Fix rare issue where pending OOO read can be left dangling if creating querier fails. #14341
• [BUGFIX] TSDB: fix check for context cancellation in LabelNamesFor. #14302
• [BUGFIX] Rules: Fix rare panic on reload. #14366
• [BUGFIX] Config: In YAML marshalling, do not output a regexp field if it was never set. #14004
• [BUGFIX] Remote-Write: reject samples with future timestamps. #14304
• [BUGFIX] Remote-Write: Fix data corruption in remote write if max_sample_age is applied. #14078
• [BUGFIX] Notifier: Fix Alertmanager discovery not updating under heavy load. #14174
• [BUGFIX] Regexes: some Unicode characters were not matched by case-insensitive comparison. #14170,#14299
• [BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515

Many thanks to the Prometheus Team and contributors:
@zenador
@jjo
@rexagod
@darshanime
@charleskorn
@fpetkovski
@carrieedwards
@colega
@pracucci
@akunszt
@DrAuYueng
@paulojmdias
@Maniktherana
@rabenhorst
@saswatamcode
@B1F030
@yeya24
@rapphil
@liam-howe-maersk
@jkroepke
@FUSAKLA
@Ranveer777

2.54.0+dedupelabels / 2024-08-09

Special release candidate build that incorporates performance improvements using
the dedupelabels Go tag. The new labels data structure deduplicates label names
and values via symbol or intern tables, trading off ~10% higher CPU usage for lower
memory usage.
Improves on 2.51.0+dedupelabels by squashing some leaks.

2.53.2 / 2024-08-09

Fix a bug where Prometheus would crash with a segmentation fault if a remote-read
request accessed a block on disk at about the same time as TSDB created a new block.

[BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515,#14523

2.54.0-rc.1 / 2024-08-05

RC1 includes a fix to a bug introduced in RC0:

• [BUGFIX] TSDB: Exclude OOO chunks mapped after compaction starts (introduced by #14396). #14584

2.54.0-rc.0+dedupelabels / 2024-08-01

Special release candidate build that incorporates performance improvements using
the dedupelabels Go tag. The new labels data structure deduplicates label names
and values via symbol or intern tables, trading off ~10% higher CPU usage for lower
memory usage.

docker pull prom/prometheus:v2.54.0-rc.0-dedupelabels

2.54.0-rc.0 / 2024-07-19

Release 2.54 brings a release candidate of a major new version of Remote Write: 2.0.
This is experimental at this time and may still change.
Remote-write v2 is enabled by default, but can be disabled via feature-flag web.remote-write-receiver.accepted-protobuf-messages.

• [CHANGE] Remote-Write: highest_timestamp_in_seconds and queue_highest_sent_timestamp_seconds metrics now initialized to 0. #14437
• [CHANGE] API: Split warnings from info annotations in API response. #14327
• [FEATURE] Remote-Write: Version 2.0 experimental, plus metadata in WAL via feature flag metadata-wal-records (defaults on). #14395,#14427,#14444
• [FEATURE] PromQL: add limitk() and limit_ratio() aggregation operators. #12503
• [ENHANCEMENT] PromQL: Accept underscores in literal numbers, e.g. 1_000_000 for 1 million. #12821
• [ENHANCEMENT] PromQL: float literal numbers and durations are now interchangeable (experimental). Example: time() - my_timestamp > 10m. #9138
• [ENHANCEMENT] PromQL: use Kahan summation for sum(). #14074,#14362
• [ENHANCEMENT] PromQL (experimental native histograms): Optimize histogram_count and histogram_sum functions. #14097
• [ENHANCEMENT] TSDB: Better support for out-of-order experimental native histogram samples. #14438
• [ENHANCEMENT] TSDB: Optimise seek within index. #14393
• [ENHANCEMENT] TSDB: Optimise deletion of stale series. #14307
• [ENHANCEMENT] TSDB: Reduce locking to optimise adding and removing series. #13286,#14286
• [ENHANCEMENT] Regexps: Optimize patterns with multiple prefixes. #13843,#14368
• [ENHANCEMENT] Regexps: Optimize patterns containing multiple literal strings. #14173
• [ENHANCEMENT] AWS SD: expose Primary IPv6 addresses as __meta_ec2_primary_ipv6_addresses. #14156
• [ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks. #10490
• [ENHANCEMENT] OpenStack SD: Use flavor.original_name if available. #14312
• [ENHANCEMENT] UI (experimental native histograms): more accurate representation. #13680,#14430
• [ENHANCEMENT] Agent: out_of_order_time_window config option now applies to agent. #14094
• [ENHANCEMENT] Notifier: Send any outstanding Alertmanager notifications when shutting down. #14290
• [ENHANCEMENT] Rules: Add label-matcher support to Rules API. #10194
• [ENHANCEMENT] HTTP API: Add url to message logged on error while sending response. #14209
• [BUGFIX] CLI: escape | characters when generating docs. #14420
• [BUGFIX] PromQL (experimental native histograms): Fix some binary operators between native histogram values. #14454
• [BUGFIX] TSDB: LabelNames API could fail during compaction. #14279
• [BUGFIX] TSDB: Fix rare issue where pending OOO read can be left dangling if creating querier fails. #14341
• [BUGFIX] TSDB: fix check for context cancellation in LabelNamesFor. #14302
• [BUGFIX] Rules: Fix rare panic on reload. #14366
• [BUGFIX] Config: In YAML marshalling, do not output a regexp field if it was never set. #14004
• [BUGFIX] Remote-Write: reject samples with future timestamps. #14304
• [BUGFIX] Remote-Write: Fix data corruption in remote write if max_sample_age is applied. #14078
• [BUGFIX] Notifier: Fix Alertmanager discovery not updating under heavy load. #14174
• [BUGFIX] Regexes: some Unicode characters were not matched by case-insensitive comparison. #14170,#14299
• [BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515

2.53.1 / 2024-07-10

This is a bug-fix release, reverting a change introduced in v2.51.0.

The bug was that remote-write would drop samples if the sending flow stalled for longer than it takes to write one "WAL segment". How long this takes depends on the data rate of your Prometheus; as a rough guide with 10 million series scraping once per minute it could be about 5 minutes. The issue is #14087.

• [BUGFIX] Remote-write: stop dropping samples in catch-up #14446

As usual, container images are available at https://quay.io/repository/prometheus/prometheus?tab=tags and https://hub.docker.com/r/prom/prometheus/tags