3.2.0-rc.0

This release contains 693 PRs from 61 authors, including new contributors Antonio Pitasi, Arpit Jain, Dylan Wylie, Felix von Platen, Geet Manghnani, J Stickler, Joseph Cheverton-Wynne, Matt Van Horn, Nabil, Oleg V. Kozlyuk, Vitaliy, Warren Amphlett, c0ff3e, mi2428, monoxane. Thank you!

Grafana Mimir version 3.2 release notes

Grafana Labs is excited to announce version 3.2 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release. For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Grafana Mimir version 3.2 includes the following key features and enhancements.

Mimir Query Engine (MQE) improvements

MQE continues to receive significant optimizations in this release:

• The experimental PromQL duration arithmetic helpers min(...) and max(...) used inside [...] range/subquery brackets and offset clauses have been renamed to min_of(...) and max_of(...).
• Add support for the experimental PromQL function histogram_quantiles.
• Add support for the native histogram trim operators </ (trim upper) and >/ (trim lower).
• Add experimental support for scalar common subexpression elimination. (see -querier.mimir-query-engine.enable-scalar-common-subexpression-elimination).
• Add experimental support for running splitting, caching and spinning off subqueries from instant queries. (see -query-frontend.use-mimir-query-engine-for-splitting-and-caching-results)
• Improve experimental support for reporting the number of samples read per query.
• Extended the "remove statically empty expressions" optimization pass to descend into subqueries.
• Support for native histograms in smoothed and anchored extended range selector modifiers.

Additional improvements

Grafana Mimir 3.2 also includes:

• Add flag to control Kafka producer compression. (-ingest-storage.kafka.producer-compression)
• Add mimirtool partition-ring remove-all-owners-and-partitions.
• Add a native histogram variant of the MimirRequestLatency.
• Add AWS Signature Version 4 (SigV4) support for shared Mimir API client commands.
• Add additional_cost_attribution_trackers configuration for multiple named cost attribution trackers per tenant.
• Add kafkatool dump find-duplicates command.
• Add shared tenant-fair compute worker pool to the ingester. (See experimental -ingester.compute-workers and -ingester.label-values-count-chunk-size)
• Add -memberlist.compression-algorithm flag to select the algorithm used to compress outgoing messages.
• Add keep-alives support to runtime configurations fetching from HTTP endpoints. (see -runtime-config.http-client-disable-keep-alives)
• Expose ingest_storage feature flag in the /api/v1/status/buildinfo endpoint.

Important changes

Grafana Mimir 3.2 introduces several updates that change default behavior and configuration. Review these changes before upgrading:

• Remote execution enabled by default. You must ensure all queriers are on Mimir 3.1 before upgrading. -query-frontend.enable-multiple-node-remote-execution-requests has been removed.
• Experimental MQE Projection Pushdown optimization pass and associated CLI flag -querier.mimir-query-engine.enable-projection-pushdown has been removed.
• Query planning metrics previously emitted with a component="querier" label are now emitted with a engine="querier" label.
• Remove the experimental prune-toggles optimization pass and its CLI flag -querier.mimir-query-engine.enable-prune-toggles. Pruning of toggled query expressions such as ... and on() (vector(0) == 1) are now handled by the remove-statically-empty-expressions optimization pass, enabled with -querier.mimir-query-engine.enable-remove-statically-empty-expressions=true.
• Hedging of query requests to ingesters is now disabled by default. The previous behaviour can be restored by setting -querier.minimize-ingester-requests-hedging-delay=3s.
• -compactor.split-and-merge-shards and -compactor.ooo-split-and-merge-shards are now rounded up to the next power of two.
• Spread the periodic idle-series cleanup across shards. (See -usage-tracker.min-time-between-shards-cleanup)
• Respect the Cache-Control: no-store request header when caching intermediate results for range vector splitting.
• Disable TCP DNS connection pooling used by memcached for service discovery by default.
• Enable query sharding by default. Disable it with -query-frontend.parallelize-shardable-queries=false

Experimental features

Grafana Mimir 3.2 includes some features that are experimental. Use these features with caution and report any issues that you encounter:

• Add experimental support for xor2 encoding. (see -ingester.float-chunk-encoding)
• Add experimental label-based access control (LBAC) for metric read queries. (see -auth.label-access-control-enabled)
• Add experimental support for splitting and caching present_over_time over range vectors in instant queries.
• Add experimental WarpStream-aware Kafka producer backend, enabled with -ingest-storage.kafka.backend=warpstream.
• Add experimental -ruler.distributor.address support for pushing rule evaluation results to distributors over native gRPC.
• Add experimental streaming label/value search HTTP endpoints. (See -querier.experimental-search-api-enabled)
• Add experimental compactor scheduler component to coordinate the work between compactors.
• Add experimental -ingest-storage.ingester-partition-metric-label-enabled flag to add ingester_partition labels to ingester's metrics.
• Add experimental -query-frontend.active-series-max-shard-concurrency to bound sub-requests concurency.
• Add experimental -query-frontend.active-series-framed-responses improving query-frontend merging performances.
• Add experimental -ingest-storage.kafka.write-timeout-overhead to cofigure the overhead added on top of the Kafka write timeout.
• Add experimental -query-frontend.subquery-spin-off-simple-subqueries to broaden subquery spin-off.

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir Helm chart is released independently. Refer to the Grafana Mimir Helm chart documentation.

Changelog

3.2.0-rc.0

Grafana Mimir

• [CHANGE] Query-frontend: -query-frontend.log-query-request-headers now rejects headers that carry credentials or session material (e.g. Authorization, Cookie, X-Api-Key) at startup, and any such headers that reach the slow-query/query-stats log paths are redacted as defense in depth. Operators that previously allow-listed such headers must remove them from the flag. #15487
• [CHANGE] Alertmanager: Upgraded the embedded prometheus/alertmanager library to v0.33.0. The per-tenant alertmanager web UI and the /-/healthy, /-/ready, /-/reload, /script.js, and /favicon.ico endpoints under the alertmanager prefix are no longer served because upstream removed the embeddable UI package. The v2 API endpoints and request/response shapes are unchanged, but a few error message strings from the upstream parser have been reformatted (for example, matcher-validation errors now include in set N). #15144, #15733
• [CHANGE] Alertmanager: The cortex_alertmanager_dispatcher_aggregation_group_limit_reached_total counter is now best-effort under concurrent alert ingest. The upstream alertmanager v0.32.0 ingests alerts via worker goroutines and the group-limit check is a racy check-then-act, so the configured limit can be exceeded under burst load. #15144
• [CHANGE] Querier: The experimental PromQL duration arithmetic helpers min(...) and max(...) used inside [...] range/subquery brackets and offset clauses have been renamed to min_of(...) and max_of(...). #15593 #15603
• [CHANGE] Querier, Store-gateway: Only send non-opaque GRPC types between queriers and store-gateways. Note that this change requires upgrading from Mimir 3.1. See associated release notes for more information. #15358
• [CHANGE] Querier: Remove experimental MQE Projection Pushdown optimization pass and associated CLI flag querier.mimir-query-engine.enable-projection-pushdown. #15618
• [CHANGE] Continuous-test: Change default values for tests.write-read-series-test.num-series and tests.write-read-series-test.max-query-age to match the values being set in jsonnet. #15705
• [CHANGE] Update Docker image bases from Debian 12 to Debian 13 (gcr.io/distroless/static-debian13; race images use base-nossl-debian13). #15629
• [CHANGE] Querier: Query planning metrics previously emitted with a component="querier" label are now emitted with a engine="querier" label instead, mirroring the similar metrics emitted by the query-frontend and other querier metrics. #15787
• [CHANGE] Querier: Reduce the default concurrency of queriers, -querier.max-concurrent, to 8. #15984
• [CHANGE] Query-frontend: the number of query shards is now always rounded up to the next power of two, both for the configured -query-frontend.query-sharding-total-shards and for the per-query computed value. #15807
• [CHANGE] Compactor: -compactor.split-and-merge-shards and -compactor.ooo-split-and-merge-shards are now rounded up to the next power of two, so compactor and query shards always mesh (one is a divisor or multiple of the other). #15807
• [CHANGE] Querier: Hedging of query requests to ingesters is now disabled by default. The previous behaviour can be restored by setting -querier.minimize-ingester-requests-hedging-delay=3s. #15976
• [CHANGE] MQE: Removed the experimental prune-toggles optimization pass and its CLI flag -querier.mimir-query-engine.enable-prune-toggles. Pruning of toggled query expressions such as ... and on() (vector(0) == 1) are now handled by the remove-statically-empty-expressions optimization pass, enabled with -querier.mimir-query-engine.enable-remove-statically-empty-expressions=true. #16037
• [CHANGE] Query-frontend and querier: enable remote execution of query plans and running sharding inside MQE by default. #16187
  • Important: When upgrading to this release, you must ensure all queriers are running Mimir 3.1 before upgrading to ensure no interruption to service.
• [CHANGE] Query-frontend and querier: multi-node remote execution is now always enabled whenever remote execution is enabled. The -query-frontend.enable-multiple-node-remote-execution-requests CLI flag and associated config file option has been removed. #16187
• [CHANGE] Query-frontend and querier: enable subset selector elimination by default. #16195
• [CHANGE] Vendored Prometheus: relabel configs now always serialize separator/replacement (upstream #18653). The /runtime_config (and ?mode=diff) output for tenants that set an empty separator/replacement in metric_relabel_configs will now show those fields explicitly. No effect on relabeling behavior. #16198
• [CHANGE] Query-frontend: Enable query sharding by default. Disable it with -query-frontend.parallelize-shardable-queries=false. #16212
• [FEATURE] Ingest storage: Add -ingest-storage.kafka.producer-compression flag to configure the Kafka producer compression codec. Supported values are none, gzip, snappy, lz4, and zstd. Set it to none to target Azure Event Hub's Kafka-compatible endpoint, which does not support compressed produce requests. #15235
• [FEATURE] Ingester: Shared tenant-fair compute worker pool. Replaces the previous per-request fanout (which could let a heavy tenant occupy all CPU) with a fixed pool of workers backed by a round-robin per-tenant queue. The label-values-cardinality endpoint is the first consumer. New experimental flags: -ingester.compute-workers (default 0 = GOMAXPROCS) and -ingester.label-values-count-chunk-size (default 32). #15493
• [FEATURE] Ingester, Block-builder: Add experimental -ingester.float-chunk-encoding flag (per-tenant float_chunk_encoding limit) to select the float chunk encoding (xor or xor2). The overrides-exporter can export it as a numeric value (4 for xor, 7 for xor2) when float_chunk_encoding is added to -overrides-exporter.enabled-metrics. #15831
• [FEATURE] Querier: Add experimental label-based access control (LBAC) for metric read queries. When enabled via -auth.label-access-control-enabled, Mimir enforces label selectors from the X-Prom-Label-Policy HTTP header at query time, filtering series and exemplars per tenant policy. Cache key isolation is applied automatically in the query-frontend. #15554
• [FEATURE] API: Add alertmanager limits (alertmanager_notification_rate_limit, alertmanager_max_dispatcher_aggregation_groups, alertmanager_max_templates_count) to the user limits API response. #15308
• [FEATURE] Mimirtool: Add AWS Signature Version 4 (SigV4) support for shared Mimir API client commands including mimirtool rules, mimirtool alertmanager, mimirtool alerts, mimirtool backfill, and mimirtool analyze ruler. #14959
• [FEATURE] Cost attribution: Support multiple named cost attribution trackers per tenant via new additional_cost_attribution_trackers config field. #15302
• [FEATURE] MQE: Add cortex_querier_inflight_query_max_age_seconds metric reporting the age of the oldest in-flight query memory consumption tracker. #15300
• [FEATURE] MQE: Add experimental support for splitting and caching present_over_time over range vectors in instant queries. #15386
• [FEATURE] Query-scheduler: Add experimental cortex_query_scheduler_queue_max_wait_seconds metric reporting how long the oldest request still waiting in the queue has been waiting since it was enqueued. Requests already dispatched to a querier for execution are excluded, so the metric reflects only queue wait time and acts as a queue-starvation detector. Reports 0 when no requests are waiting. Enabled by default; can be disabled with -query-scheduler.queue-max-wait-metric-enabled=false. #15419 #15960
• [FEATURE] Query-scheduler: Add experimental cortex_query_scheduler_max_queue_length metric reporting the per-tenant peak queue length observed since the last scrape. Enable with -query-scheduler.max-queue-length-metric-enabled=true. #15906
• [FEATURE] MQE: Add support for experimental PromQL functions min_of and max_of. #15597
• [FEATURE] MQE: Add support for the experimental PromQL function histogram_quantiles, which computes multiple quantiles from classic or native histograms in a single call. #15710
• [FEATURE] MQE: Add support for the native histogram trim operators </ (trim upper) and >/ (trim lower), including query sharding support. #15708 #15711
• [FEATURE] Ingester: Add experimental early compaction of non-owned series. After a ring change, the ingester tracks series whose ownership shifted to another replica and flushes them into a block, evicting them from the TSDB head ahead of the regular head-compaction cycle. Eviction triggers when both (a) the in-memory series count exceeds the local threshold derived from -ingester.early-head-compaction-owned-series-threshold and the -ingester.early-compaction-non-owned-series-min-grace-period has elapsed, or (b) the -ingester.early-compaction-non-owned-series-max-grace-period has elapsed, regardless of the threshold. Disabled by default; enable with -ingester.early-compaction-non-owned-series-enabled. Requires -ingester.track-ingester-owned-series or -ingester.use-ingester-owned-series-for-limits to be enabled. Adds the cortex_ingester_tsdb_early_compaction_non_owned_series_triggered_total counter. #15314 #15653 #15657 #15661
• [FEATURE] Ingester: Add experimental cortex_ingester_tsdb_head_chunks_max_mmapped gauge reporting the maximum, across all per-tenant TSDBs, of the maximum number of head chunks memory-mapped for any individual series during the last memory-mapping pass. Temporary measurement metric; will be removed once we have collected enough data. #15616
• [FEATURE] kafkatool: Add dump find-duplicates command to scan an exported dump and report float samples re-sent with the same timestamp and value as the previous sample for a series. These exact duplicates are silently dropped by the ingester but still count toward received-samples metrics. An optional --tenant flag restricts the scan to a single tenant. #15506
• [FEATURE] Ingest storage: Add an experimental WarpStream-aware Kafka producer backend, enabled with -ingest-storage.kafka.backend=warpstream. It hedges and reroutes produce requests across WarpStream agents to reduce write tail latency, and is tuned via the experimental -ingest-storage.kafka.warpstream-* flags. #15236 #15809
• [FEATURE] MQE: Add experimental support for running splitting, caching and spinning off subqueries from instant queries inside MQE. Enabled with -query-frontend.use-mimir-query-engine-for-splitting-and-caching-results=true. #15348 #15393 #15397 #15650 #15720 #15750 #15769 #15783 #15787 #15795 #15804 #15822 #15823 #15846 #15827 #15884 #15887 #15912 #15928 #15930 #15954 #15955 #15978 #15999 #16015 #16213
  • Important: When splitting and caching inside MQE is enabled, the -query-frontend.query-sharding-max-sharded-queries limit applies per time-split interval and spun-off subquery, rather than to the entire interval (or entire spun-off subquery).
• [FEATURE] Ruler: Add experimental -ruler.distributor.address support for pushing rule evaluation results to distributors over native gRPC instead of using an internal distributor. Configure the per-request timeout with -ruler.distributor.remote-timeout. #15891
• [FEATURE] MQE: Add experimental support for scalar common subexpression elimination. Enabled with -querier.mimir-query-engine.enable-scalar-common-subexpression-elimination=true. #15991
• [FEATURE] Compactor scheduler: Add the experimental compactor scheduler component, which coordinates work between compactors and exposes additional metrics about pending and active compaction work. #14493 #14553 #14657 #14747 #14768 #14772 #14781 #14804 #14833 #14844 #14849 #14910 #14937 #14945 #14987 #15026 #15094 #15112 #15113 #15267 #15293 #15321 #15606 #15728 #15850 #15897 #15974 #15989 #16042 #16099
• [FEATURE] Mimirtool: Add mimirtool partition-ring remove-all-owners-and-partitions to forcefully remove all owners and partitions from a partition ring in a single operation. #16081
• [FEATURE] Ingester: added experimental -ingest-storage.ingester-partition-metric-label-enabled flag. When set together with -ingest-storage.enabled, every metric emitted by the ingester (and the ingest-storage reader / partition-ring lifecycler it owns) carries an ingester_partition label identifying the Kafka partition the ingester consumes. Planned to default to enabled in Mimir 3.2 and to be removed in Mimir 3.5. #15130
• [FEATURE] Querier: experimental streaming label/value search HTTP endpoints /api/v1/search/{metric_names,label_names,label_values}. Gated by -querier.experimental-search-api-enabled (default false). #15233, #15349, #15301, #15347, #15364
• [FEATURE] Query-tee: Add per-backend exclude_tenants option to the -backend.config-file configuration. Requests whose X-Scope-OrgID tenants are all excluded are not sent to that backend, while the preferred backend always receives all traffic. #16075
• [ENHANCEMENT] Query-frontend: Add experimental -query-frontend.active-series-max-shard-concurrency to bound how many sharded active series (and active native histogram metrics) sub-requests are dispatched and merged concurrently within a single request, limiting the resource usage caused by fanning out to a large number of shards, both on queriers and on the query-frontend. 0 (the default) keeps the previous unbounded behavior. #15970
• [ENHANCEMENT] Query-frontend, Querier: Add experimental -query-frontend.active-series-framed-responses to request active series responses from queriers in a length-delimited framed format that the query-frontend can merge using significantly less CPU (roughly 5-6x faster merges for high shard counts). Queriers that don't support the format fall back to JSON transparently. #15971
• [ENHANCEMENT] Store-gateway, Ingester: Add read support for XOR2 chunk encoding. XOR2 is a new Prometheus TSDB encoding that provides better compression than XOR, particularly for stale markers. #15371
• [ENHANCEMENT] MQE: Improve experimental support for reporting the number of samples read per query. #14838 #15179 #15191 #15220 #15223 #15232 #15237 #15255 #15276 #15282 #15285
• [ENHANCEMENT] Distributor: Relabel middleware returns early if neither label dropping nor relabeling is configured. #15246
• [ENHANCEMENT] Distributor: Improve distributor push middleware cleanup handling. #15245
• [ENHANCEMENT] Distributor: Avoid allocating a string per received timeseries when aggregating cost attribution samples per attribution group in the write path. #15751
• [ENHANCEMENT] Ingest storage: Reject the whole batch of records of a Kafka write call when the configured -ingest-storage.kafka.producer-max-buffered-bytes limit is reached, instead of rejecting individual records. #15227
• [ENHANCEMENT] MQE: Simplify unless and or operations where one side can be proven to be empty by inspecting the expression. #15198
• [ENHANCEMENT] Store-gateway: Remove outdated limit on caching LabelValues responses that contain more than 655360 values. The gob library panic which required workaround was fixed. #5021 #15271
• [ENHANCEMENT] MQE: Reduce memory consumption of range vector splitting when many consecutive intervals are not cached. #15173
• [ENHANCEMENT] MQE: Allow common subexpression elimination to deduplicate range vector splitting nodes when subset selector elimination has merged the selectors. #16215
• [ENHANCEMENT] MQE: Reduce the number of requests to the intermediate result cache of range vector splitting by using the batch API. #16024
• [ENHANCEMENT] Querier: track physical and equivalent samples read for remote read requests in query statistics, mirroring the statistics emitted for instant and range queries. #15694
• [ENHANCEMENT] Ingest storage: Add cortex_ingest_storage_writer_serialize_duration_seconds native histogram metric tracking the time spent serializing an incoming request to Kafka records. #15527
• [ENHANCEMENT] Memberlist: Add -memberlist.compression-algorithm flag to select the algorithm used to compress outgoing messages. Supported values: lzw (default) and snappy. The flag is ignored when -memberlist.compression-enabled is false. Before reconfiguring any node to emit a new algorithm, upgrade every cluster member to a build that can decode it, otherwise messages are dropped. #15357
• [ENHANCEMENT] Memberlist: TCP push-pull and other TCP stream messages now skip compression when the compressed output is no smaller than the input, falling back to a plaintext frame. Mirrors existing UDP behaviour; receivers continue to decode both compressed and plaintext frames so the change is wire-compatible. #15357
• [ENHANCEMENT] Memberlist: Reduce per-call allocations on the compression and TCP state-sync receive paths via internal buffer pools. #15357
• [ENHANCEMENT] Memberlist: Add memberlist.processed-messages-queue-size flag to set the size of the per-key internal queue for processing messages received from other nodes. Increasing this value may help to avoid dropping per-key updates when the node is processing many updates for the same key. #15536
• [ENHANCEMENT] MQE: Respect the Cache-Control: no-store request header when caching intermediate results for range vector splitting. #15148
• [ENHANCEMENT] MQE: Extend experimental support for computing multiple aggregations over the same data without buffering to quantile aggregations. #15624
• [ENHANCEMENT] Ingest storage: skip per-record tracing span and attribute allocations on the Kafka fetch path when the producer trace is not sampled. The producer's trace context is still extracted from record headers for every record. #15614
• [ENHANCEMENT] Ingest storage: the experimental WarpStream Kafka producer backend (-ingest-storage.kafka.backend=warpstream) now traces produce requests, emitting the same producer spans and traceparent propagation as the default Kafka backend. #16039
• [ENHANCEMENT] Distributor: Add a tracing span around the OTLP to Prometheus conversion so its latency is independently visible in traces. #15682
• [ENHANCEMENT] Runtimeconfig: The HTTP client used to fetch runtime configurations from HTTP endpoints now has keep-alives disabled by default. New CLI flag -runtime-config.http-client-disable-keep-alives is enabled by default, an can be set to false in-order to re-enable keep-alives. #15695
• [ENHANCEMENT] MQE: Support for native histograms in smoothed and anchored extended range selector modifiers. #15398
• [ENHANCEMENT] Usage-tracker: Spread the periodic idle-series cleanup across shards with a configurable minimum delay between shards (default 25ms), gated behind -usage-tracker.min-time-between-shards-cleanup, to avoid blocking latency-sensitive series-tracking calls on large single-tenant instances. #15871
• [ENHANCEMENT] Usage-tracker, distributor: Add experimental synchronous batched tracking. When -distributor.usage-tracker-client.use-sync-batched-tracking is enabled, synchronous series-tracking calls linger for up to -distributor.usage-tracker-client.sync-batch-delay and are sent together in a single batch RPC, reducing the number of network calls while still returning rejected series to each caller. By default all partitions flush together on a shared timer so the usage-tracker can coalesce the packets; set -distributor.usage-tracker-client.sync-batch-independent-partition-timeouts to make each partition linger independently instead. #15805
• [ENHANCEMENT] Usage-tracker: Add admin debug pages, with tracker store stats. #15882
• [ENHANCEMENT] Mimir: Expose ingest_storage feature flag in the /api/v1/status/buildinfo endpoint, reflecting whether Mimir runs with ingest storage architecture. #15743
• [ENHANCEMENT] Block-builder: Respect -blocks-storage.tsdb.bigger-out-of-order-blocks-for-old-samples to produce 24h blocks for out-of-order data belonging to previous days. #15892
• [ENHANCEMENT] MQE: Ensure that intermediate results cache keys can not exceed the cache backend's key-size limit when a query federates over many tenants. #15847
• [ENHANCEMENT] Query-frontend: add a retries field to the "query stats" log line reporting the number of times requests were retried while processing the query. The value is 0 when all requests succeeded on their first attempt. #15929
• [ENHANCEMENT] Query-frontend: Add query-frontend.cardinality-sharding-max-sharded-queries to optionally limit sharding for cardinality/active_series and cardinality/active_native_histogram_metrics endpoints separately from query-frontend.query-sharding-max-sharded-queries. #15922
• [ENHANCEMENT] MQE: Extended the "remove statically empty expressions" optimization pass to descend into subqueries. Previously it skipped subqueries. #16038
• [ENHANCEMENT] Ingest storage: Add experimental -ingest-storage.kafka.write-timeout-overhead to configure the overhead added on top of the Kafka write timeout (default 2s, unchanged). #16023
• [ENHANCEMENT] MQE: Use series selected for one side to reduce data selected on the other side in binary operations that use ignoring or no on/ignoring clause. #15178
• [ENHANCEMENT] Query-frontend: Add experimental flags to broaden subquery spin-off. -query-frontend.subquery-spin-off-simple-subqueries=true spins off subqueries whose inner expression was previously considered too simple to spin off and -query-frontend.subquery-spin-off-with-excess-downstream-queries=true spins off subqueries when the rewritten query contains more downstream queries than spun-off subqueries. All are disabled by default and require subquery spin-off to be enabled with -query-frontend.subquery-spin-off-enabled=true. #16211
• [ENHANCEMENT] Block-builder: Disable authentication for AssignJob and UpdateJob gRPC methods as they are control-plane methods that don't have an inherent orgID. #16222
• [ENHANCEMENT] gRPC clients: Add advanced CLI flags to configure client keepalive: #16221
  • -<prefix>.keepalive-time (default 20s)
  • -<prefix>.keepalive-timeout (default 10s)
• [BUGFIX] Continuous-test: Fix native histogram queries being subject to the PromQL lookback period, which carried the most recently written histogram forward past its real timestamp. This caused startup recovery of the last written histogram sample to always fail (the recovery query end time is "now", so lookback returned samples whose values didn't match the expected value for those later timestamps), silently abandoning the histogram write history on every restart. Histogram queries now wrap the selector in last_over_time(...[1s]), matching the existing float query behavior. #16163
• [BUGFIX] Memberlist: Validate experimental propagation delay tracker config at startup so a non-positive beacon interval or lifetime fails with a clear error instead of panicking when the tracker is enabled. #16221
• [BUGFIX] Query-frontend: Fix cardinality_analysis_max_results being ignored when set higher than the default of 500. #15581
• [BUGFIX] Ingest storage: Fix KafkaProducer.ProduceSync() returning a single result with a nil record when the context is canceled, instead of one result per input record (with the record set) as the underlying franz-go client does. #15199
• [BUGFIX] Ingest storage: Fix the partition reader skipping available records when replaying from the max replay period with file-based offset enforcement enabled. If the timestamp-based start offset lookup unexpectedly resolves to the partition end despite a recent tail record, the reader retries the lookup and then clamps the start offset to the partition start. #16161
• [BUGFIX] Ingest storage: Fix cortex_ingest_storage_reader_receive_delay_seconds inflation by no longer setting the Kafka record Timestamp on the distributor side; the Kafka client now sets it at produce time. #15572
• [BUGFIX] Distributor: Return HTTP 200 with OTLP partial-success when only some samples in an OTLP request are rejected by distributor-level validation (e.g. too_far_in_past). #15253
• [BUGFIX] MQE: Bugfixes for experimental range vector splitting. #15147 #15270 #14878
• [BUGFIX] Querier: Fix querier ScaledObjects native histogram querying and triggering MimirAutoscalerKedaFailing when queriers have no traffic because cortex_querier_request_duration_seconds_sum is not published until the first request is received. #15106
• [BUGFIX] Fix build failure on Windows and FreeBSD due to reference leaks instrumentation code. Enabling reference leaks instrumentation in those platforms now causes a configuration validation error instead. #15291
• [BUGFIX] Query-frontend: Fixed a memory leak caused that could occur on some error paths if MQE was enabled. #15392
• [BUGFIX] MQE: Fix issue where subqueries unnecessarily compute and then discard an additional step if the parent query is not aligned to the step. #15438
• [BUGFIX] Upgrade Go to 1.26.4 to address CVE-2026-42507. #15566
• [BUGFIX] Upgrade Go to 1.26.5 to address CVE-2026-39822 and CVE-2026-42505. #16066
• [BUGFIX] Memcached: Disable TCP DNS connection pooling used for service discovery by default. #15573
• [BUGFIX] Ingest storage: Fix cortex_ingest_storage_writer_produce_records_enqueued_total not being incremented when KafkaProducer.ProduceSync() rejects a batch because a record has its Timestamp set by the caller. #15610
• [BUGFIX] Compactor: Remove temporary block upload validation directories left behind in the data directory when the compactor crashes mid-validation. This prevents leaking disk space. #15647
• [BUGFIX] Continuous-test: Fix a crash when histogram tests were enabled in combination with the OTLP-HTTP write protocol. #15641
• [BUGFIX] Store-gateway: Fix a regex label matcher that also matches the empty string (e.g. label=~"|foo|bar") incorrectly excluding series that don't have the label, potentially resulting in incomplete query results. #15767
• [BUGFIX] Block-builder-scheduler: Exit cleanly when shut down during startup observation. #15730
• [BUGFIX] Ingest storage: Cap maximum Kafka protocol version, the client negotiates with the broker to v3.9.0. #15745
• [BUGFIX] MQE: Report a query that panics during evaluation as failed in the evaluation stats log, instead of logging it as successful. The querier still re-panics afterwards, crash behaviour is unchanged. #15753
• [BUGFIX] Memcached: Fix issue where cache-related trace spans included events emitted with an empty name label. #15794
• [BUGFIX] MQE: Fix issue where LBAC is not respected by range vector splitting cache. #15802
• [BUGFIX] Block-builder-scheduler: Fix a spurious "time went backwards" warning logged at startup when a partition has no records after the scan time. #15855
• [BUGFIX] Compactor: Fix GatherBlockHealthStats postings walk error check to prevent swallowing errors. #15895
• [BUGFIX] MQE: Don't evaluate unnecessary range vector splitting ranges when a split range vector is part of a spun-off subquery and running time-splitting and caching inside MQE is enabled. #15931
• [BUGFIX] MQE: Fix info() function incorrectly handling negated name matchers. #15168
• [BUGFIX] MQE: Fix this indicates something has been returned to a pool more than once panic when a sum() or avg() group contains, at the same output step, a float sample and native histograms that cannot be added together (e.g. exponential and custom bucket schemas). #16059
• [BUGFIX] Query-frontend: Fix issue where series for a range query can be returned in the wrong order if splitting applies and splitting is not running inside MQE. #16036
• [BUGFIX] Querier: Fix issue where exemplars can be returned in the wrong order if a series contains a label that is a prefix of another (eg. env="foo" and env="foobar"). #16036
• [BUGFIX] Querier: Fix experimental search /api/v1/search/metric_names?include_metadata=true almost never returning metric metadata. #16062, #16214
• [BUGFIX] Querier: Stop querying a partition that has been inactive for longer than -querier.query-ingesters-within, preventing query failures when the partition is still registered but has no available ingesters to serve the queries. #15721
• [BUGFIX] Query-frontend: Fix queue_time_seconds in the query stats log always reporting 0 when a query is cancelled while still waiting in the query-scheduler queue. #16094
• [BUGFIX] MQE: Fix the binary operation narrow-selectors optimization incorrectly using binary operation matchers across an on() / on(...) group_left/group_right join boundary, which could cause some queries to unexpectedly evaluate as an empty result. #16155
• [BUGFIX] Packaging: Fix the DEB/RPM packages shipping the mimir, mimirtool, metaconvert, and query-tee binaries without the executable bit set, which caused mimir.service to fail to start. #16166
• [BUGFIX] Query-frontend: Fix a goroutine leak when a querier's streaming response arrives just as the query is cancelled: the goroutine handling the response could stay blocked forever writing a response body that would never be read. #16151
• [BUGFIX] MQE: Fix issue where a sentinel value was inadvertently returned to a pool where it could be mutated by multiple threads at once. #16205

Mixin

• [CHANGE] Dashboards: Show maximum queue length, not minimum queue length, on the "Queue length" panel in the "Query-scheduler" row of the "Reads" and "Remote ruler reads" dashboards. #15326
• [CHANGE] Alerts: MimirIngesterKafkaReadFailed now fires as warning after 5m, and escalates to critical if it persists for 30m. MimirStrongConsistencyEnforcementFailed severity changed from critical to warning, since queriers retry on a different ingester and fire their own alerts if that retry fails. #16019
• [CHANGE] Remove Grafana Enterprise Metrics (GEM) specific build of the mixin. #16031
• [ENHANCEMENT] Alerts: Make MimirInconsistentRuntimeConfig alert less flaky when performing multiple configuration changes in a row in a large Kubernetes cluster. #15257
• [ENHANCEMENT] Alerts: Widen the MimirBlockBuilderPersistentJobFailure lookback window to 20m to prevent the alert from flapping. #15332
• [ENHANCEMENT] Alerts: Add a native histogram variant of the MimirRequestLatency alert, distinguished by the histogram label (classic or native). #15413
• [ENHANCEMENT] Alerts: Add a "for" period to MimirBucketIndexNotUpdated to avoid false-positives when a compactor restarted near the end of the previous update cycle. #15935
• [ENHANCEMENT] Dashboards: Add "Rejected queries rate" panel to the Tenants dashboard showing the per-tenant rate of queries rejected by the query-frontend, split by reason. #14979
• [ENHANCEMENT] Dashboards: Add 100th percentile to query expression percentiles graph. #15421
• [ENHANCEMENT] Dashboards: Add the experimental streaming search API endpoints to the "Overview" per-endpoint query breakdown, and include the ingester SearchLabelNames/SearchLabelValues gRPC routes in the ingester panels of the "Reads", "Queries", and "Remote ruler reads" dashboards. #15571
• [ENHANCEMENT] Dashboards: Add "p90 compaction delay by level" and "Store-gateway blocks queried by level" panels to the "Compaction" row of the "Compactor" dashboard. #15619
• [ENHANCEMENT] Store-gateway, Querier: Push down the limit parameter of the /prometheus/api/v1/series endpoint to store-gateways, so they stop loading series (and their chunks) once the limit is reached instead of fetching all matching series and discarding the excess downstream. #15834
• [ENHANCEMENT] Dashboards: Split the server-side "Usage Tracker" row of the "Writes" dashboard into separate "TrackSeries" (non-batched) and "TrackSeriesBatch" (batched) rows, so batched tracking RPCs are visible now that synchronous batched tracking can drive TrackSeriesBatch. #15805
• [ENHANCEMENT] Dashboards: Simplify the ingest storage produced-records queries in the "Writes" and "Ruler" dashboards by removing the fallback to the cortex_ingest_storage_writer_produce_requests_total and cortex_ingest_storage_writer_produce_failures_total metrics, which were renamed to cortex_ingest_storage_writer_produce_records_enqueued_total and cortex_ingest_storage_writer_produce_records_failed_total more than a year ago. #16035
• [ENHANCEMENT] Dashboards: Add optional per-zone panels for multi-zone write path deployments to the "Writes" dashboard, enabled via the show_multi_zone_write_path_panels config option (disabled by default). When enabled, the gateway and distributor "Requests / sec" and "Kafka produced records / sec" panels break down the traffic by availability zone, and "Latency per zone" panels are added next to the aggregate latency panels. #16206
• [BUGFIX] Dashboards: Fix the classic/ingest-storage split in the "Tenants", "Top tenants" and "Writes" dashboards so that selecting multiple clusters with a mix of architectures no longer drops the classic clusters' data. The unless on (job) filter against cortex_partition_ring_partitions now also matches on the cluster aggregation labels. #15400
• [BUGFIX] Alerts: Update MimirRulerInstanceHasNoRuleGroups to not alert on false-positives when rulers are running in multiple zones. #16029

Jsonnet

• [CHANGE] Query-frontend: Increase default query-frontend cache size limit to 25MB. #14857
• [CHANGE] Query-frontend: Increase memory requested and limit to 2GiB and 4GiB respectively. #15688
• [CHANGE] Continuous-test: Don't explicitly set tests.write-read-series-test.num-series and tests.write-read-series-test.max-query-age to their default values. #15705
• [CHANGE] Request 50Mi of ephemeral storage for the distributor, ingester, querier, query-frontend, query-scheduler, ruler-querier, ruler-query-frontend, ruler-query-scheduler, store-gateway, compactor, compactor-scheduler and continuous-test containers. Configure with the new ephemeral_storage_request_size option (set it to null to disable). #15916
• [CHANGE] Querier: Reduce the default concurrency of queriers, -querier.max-concurrent, to 8. #15984
• [CHANGE] Query-frontend: Enable query sharding by default. Disable it by setting _config.query_sharding_enabled to false. #16212
• [FEATURE] Compactor: add support for deploying the experimental compactor-scheduler. Enable with compactor_scheduler_enabled: true. #15850
• [FEATURE] Compactor: add experimental compactor autoscaling, enabled with autoscaling_compactor_enabled: true. When the compactor-scheduler is enabled, compactors are autoscaled based on the estimated time to drain the scheduler queue instead of CPU utilization. #15850
• [ENHANCEMENT] Updated rollout-operator jsonnet library to v0.38.1. #15328, #15626, #16129
• [ENHANCEMENT] Make range vector splitting configurable per query path. #15706
• [ENHANCEMENT] Add newMimirtoolBlocksJob and subcommand-specific helpers to run mimirtool blocks as Kubernetes Jobs. #15757
• [BUGFIX] Continuous-test: Include ._config.commonConfig in arguments passed to continuous-test. #15988

Documentation

• [ENHANCEMENT] Update MimirRingMembersMismatch runbook: check for spurious ingesters left in the ring after a scale down. #16169

Tools

• [CHANGE] The mark-blocks, listblocks, copyblocks, splitblocks, and undelete-blocks tools are now subcommands of mimirtool blocks: mark, list, copy, split, undelete. #15757
• [FEATURE] Copyblocks: add support for the block upload API as a copy destination. #15330
• [ENHANCEMENT] Mimirtool: partition-ring subcommands now accept an optional --partition-ring.key flag to select the KV store key of the partition ring to operate on. It defaults to ingester-partitions. #15719
• [ENHANCEMENT] Makefile: build-mixin and mixin-screenshots can now be configured to use native histograms for latency panels in dashboards. #15269
• [ENHANCEMENT] kafkatool: Add a README. #15898

All changes in this release: mimir-3.1.4...mimir-3.2.0-rc.0