v0.23.2-rc.0

Fixed

• #4795 Query: Fix deadlock in endpointset.

v0.24.0-rc.0

Added

• #4228 Tools thanos bucket inspect: Add flag --output to provide output method (table,csv,tsv).
• #4282 Query: breaking ⚠️ Add --endpoint flag to the querier. The --store flag will eventually be replaced.
• #4680 Query: Add exemplar.partial-response flag to control partial response.
• #4679 Query: Add enable-feature flag to enable negative offsets and @ modifier, similar to Prometheus.
• #4696 Query: Add cache name to tracing spans.
• #4710 Store: Add metric to capture timestamp of the last loaded block.
• #4736 S3: Add capability to use custom AWS STS Endpoint.
• #4764 Compact: Add block-viewer.global.sync-block-timeout flag to set the timeout of synchronization block metas.
• #4801 Compact: Add Prometheus metrics for tracking the progress of compaction and downsampling.
• #4444 UI: Add mark deletion and no compaction to the Block UI.
• #4576 UI: Add filter compaction level to the Block UI.
• #4731 Rule: Add stateless mode to ruler.
• #4612 Sidecar: Add --prometheus.http-client and --prometheus.http-client-file flag for sidecar to connect to Prometheus with basic auth or TLS.
• #4847 Query: Add --alert.query-url which is used in the UI for rules/alerts pages. By default the HTTP listen address is used for this URL.
• #4856 Mixin: Add Query Frontend to Grafana dashboard.
• #4874 Query: Add --endpoint-strict flag to statically configure Thanos API server endpoints. It is similar to --store-strict but supports passing any Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI.
• #4868 Rule: Support ruleGroup limit introduced by Prometheus v2.31.0.
• #4897 Query: Add validation for querier address flags.

Fixed

• #4508 Sidecar, Mixin: Rename ThanosSidecarUnhealthy to ThanosSidecarNoConnectionToStartedPrometheus; Remove ThanosSidecarPrometheusDown alert; Remove unused thanos_sidecar_last_heartbeat_success_time_seconds metrics.
• #4663 Fetcher: Fix discovered data races.
• #4754 Query: Fix possible panic on stores endpoint.
• #4753 Store: validate block sync concurrency parameter.
• #4779 Examples: Fix the interactive test for MacOS users.
• #4792 Store: Fix data race in BucketedBytes pool.
• #4769 Query Frontend: Add "X-Request-ID" field and other fields to start call log.
• #4709 Store: Fix panic when the application is stopped.
• #4777 Query: Fix data race in exemplars server.
• #4811 Query: Fix data race in metadata, rules, and targets servers.
• #4795 Query: Fix deadlock in endpointset.

Changed

• #4864 UI: Remove the old PromQL editor.
• #4708 Receive: Remove gRPC message size limit, which fixes errors commonly seen when receivers forward messages within a hashring.

v0.23.1

Fixed

• #4714 EndpointSet: Do not use unimplemented yet new InfoAPI to obtain metadata (avoids unnecessary HTTP roundtrip, instrumentation/alerts spam and logs)

v0.23.0

Thanks to all 44 contributors, who had committed good work to this release! 🤗

@GiedriusS @bwplotka @Nexucis @Namanl2001 @hanjm @clyang82 @Hangzhi @pracucci @metalmatze @iamrajiv @roystchiang @saswatamcode @adzshaf @Abhishek357 @Bacto @AnaisUrlichs @2pk03 @vanugrah @arajkumar @galan @ethernoy @hitanshu-mehta @ianbillett @ilangofman @jesusvazquez @jmichalek132 @OGKevin @matej-g @56quarters @nishidhaSri @philipgough @phillebaba @JRavi2 @crsandeep @simonpasquier @thanatchakromsang @weyert @wiardvanrij @huyan0 @yangyang919 @zhyChesterCheung @rhassanein @songjiayang @yahaa

Notable changes:

• Query-frontend overhead reduced 20% (change your max_idle_conns_per_host to at least 100!)
• Massive enhancement over Block UI: Search capabilities, overlap finder and more!
• Added Baidu BOS object storage support
• Improved tracing metadata. It now shows exact series, samples and overall bytes processes per component in the path of StoreAPI.Series query.
• Easier use of non Store APIs: auto-discovery through endpoint flag. Old components will work as they were, no action is required unless you want to explicitly disable certain APIs (e.g rules, exemplars, metadata etc) if you don't want them.
• Added few Prometheus PromQL functions (upgraded Prometheus deps)

Added

• #4453 Tools thanos bucket web: Add flag --selector.relabel-config-file / --selector.relabel-config / --max-time / --min-time to filter served blocks.
• #4482 Store: Add http_config option for COS object store client.
• #4487 Query/Store: Add memcached auto discovery support for all caching clients.
• #4444 UI: Add search to the Block UI.
• #4509 Logging: Add duration_ms in int64 to the logs for easier log filtering.
• #4462 UI: Highlighting blocks overlap in the Block UI.
• #4469 Compact: Add flag compact.skip-block-with-out-of-order-chunks to skip blocks with out-of-order chunks during compaction instead of halting.
• #4506 Store: Add Baidu BOS object storage, see documents for further information.
• #4552 Compact: Add thanos_compact_downsample_duration_seconds histogram metric.
• #4594 Reloader: Expose metrics in config reloader to give info on the last operation.
• #4619 Tracing: Added consistent tags to Series call from Querier about number important series statistics: processed.series, processed.samples, processed.samples and processed.bytes. This will give admin idea of how much data each component processes per query.
• #4623 Query-frontend: Make HTTP downstream tripper (client) configurable via parameters --query-range.downstream-tripper-config and --query-range.downstream-tripper-config-file. If your downstream URL is localhost or 127.0.0.1 then it is strongly recommended to bump max_idle_conns_per_host to at least 100 so that query-frontend could properly use HTTP keep-alive connections and thus reduce the latency of query-frontend by about 20%.

Fixed

• #4468 Rule: Fix temporary rule filename composition issue.
• #4476 UI: Fix incorrect html escape sequence used for '>' symbol.
• #4532 Mixin: Fix "all jobs" selector in thanos mixin dashboards.
• #4607 Azure: Fix Azure MSI Rate Limit.

Changed

• #4519 Query: Switch to miekgdns DNS resolver as the default one.
• #4586 Update Prometheus/Cortex dependencies and implement LabelNames() pushdown as a result; provides massive speed-up for the labels API in Thanos Query.
• #4421 breaking ⚠️: --store (in the future, to be renamed to --endpoints) now supports passing any APIs from Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI (in oppose in the past you have to put it in hidden --targets, --rules etc flags). --store will now automatically detect what APIs server exposes.
• #4669 Moved Prometheus dependency to v2.30.

v0.23.0-rc.1

Fixes couple of critical bugs vs rc.0 (:

Added

• #4453 Tools thanos bucket web: Add flag --selector.relabel-config-file / --selector.relabel-config / --max-time / --min-time to filter served blocks.
• #4482 Store: Add http_config option for COS object store client.
• #4487 Query/Store: Add memcached auto discovery support for all caching clients.
• #4444 UI: Add search to the Block UI.
• #4509 Logging: Add duration_ms in int64 to the logs for easier log filtering.
• #4462 UI: Highlighting blocks overlap in the Block UI.
• #4469 Compact: Add flag compact.skip-block-with-out-of-order-chunks to skip blocks with out-of-order chunks during compaction instead of halting.
• #4506 Store: Add Baidu BOS object storage, see documents for further information.
• #4552 Compact: Add thanos_compact_downsample_duration_seconds histogram metric.
• #4594 Reloader: Expose metrics in config reloader to give info on the last operation.
• #4619 Tracing: Added consistent tags to Series call from Querier about number important series statistics: processed.series, processed.samples, processed.samples and processed.bytes. This will give admin idea of how much data each component processes per query.
• #4623 Query-frontend: Make HTTP downstream tripper (client) configurable via parameters --query-range.downstream-tripper-config and --query-range.downstream-tripper-config-file. If your downstream URL is localhost or 127.0.0.1 then it is strongly recommended to bump max_idle_conns_per_host to at least 100 so that query-frontend could properly use HTTP keep-alive connections and thus reduce the latency of query-frontend by about 20%.

Fixed

• #4468 Rule: Fix temporary rule filename composition issue.
• #4476 UI: Fix incorrect html escape sequence used for '>' symbol.
• #4532 Mixin: Fix "all jobs" selector in thanos mixin dashboards.
• #4607 Azure: Fix Azure MSI Rate Limit.

Changed

• #4519 Query: Switch to miekgdns DNS resolver as the default one.
• #4586 Update Prometheus/Cortex dependencies and implement LabelNames() pushdown as a result; provides massive speed-up for the labels API in Thanos Query.
• #4421 breaking ⚠️: --store (in the future, to be renamed to --endpoints) now supports passing any APIs from Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI (in oppose in the past you have to put it in hidden --targets, --rules etc flags). --store will now automatically detect what APIs server exposes.
• #4669 Moved Prometheus dependency to v2.30.

v0.23.0-rc.0

Added

• #4453 Tools thanos bucket web: Add flag --selector.relabel-config-file / --selector.relabel-config / --max-time / --min-time to filter served blocks.
• #4482 Store: Add http_config option for COS object store client.
• #4487 Query/Store: Add memcached auto discovery support for all caching clients.
• #4444 UI: Add search to the Block UI.
• #4509 Logging: Add duration_ms in int64 to the logs for easier log filtering.
• #4462 UI: Highlighting blocks overlap in the Block UI.
• #4469 Compact: Add flag compact.skip-block-with-out-of-order-chunks to skip blocks with out-of-order chunks during compaction instead of halting.
• #4506 Store: Add Baidu BOS object storage, see documents for further information.
• #4552 Compact: Add thanos_compact_downsample_duration_seconds histogram metric.
• #4594 Reloader: Expose metrics in config reloader to give info on the last operation.
• #4619 Tracing: Added consistent tags to Series call from Querier about number important series statistics: processed.series, processed.samples, processed.samples and processed.bytes. This will give admin idea of how much data each component processes per query.
• #4623 Query-frontend: Make HTTP downstream tripper (client) configurable via parameters --query-range.downstream-tripper-config and --query-range.downstream-tripper-config-file. If your downstream URL is localhost or 127.0.0.1 then it is strongly recommended to bump max_idle_conns_per_host to at least 100 so that query-frontend could properly use HTTP keep-alive connections and thus reduce the latency of query-frontend by about 20%.

Fixed

• #4468 Rule: Fix temporary rule filename composition issue.
• #4476 UI: Fix incorrect html escape sequence used for '>' symbol.
• #4532 Mixin: Fix "all jobs" selector in thanos mixin dashboards.
• #4607 Azure: Fix Azure MSI Rate Limit.

Changed

• #4519 Query: Switch to miekgdns DNS resolver as the default one.
• #4586 Update Prometheus/Cortex dependencies and implement LabelNames() pushdown as a result; provides massive speed-up for the labels API in Thanos Query.
• #4421: breaking ⚠️: --store (in the future, to be renamed to --endpoints) now supports passing any APIs from Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI (in oppose in the past you have to put it in hidden --targets, --rules etc flags). --store will now automatically detect what APIs server exposes.

v0.22.0

Highlights

• Added offline deduplication that helps saving even more space on the remote object storage & increases query performance
• The receive dual-mode has been implemented
• Receive now is able to ingest exemplars and it is possible to query them
• Receive now uses a few percent less of CPU time with the same load
• Azure object storage has been refactored to support MSI authentication & it's now much more configurable

This release contains improvements from 28 authors (git log --pretty="%ae" origin/release-0.21...origin/release-0.22 | sort | uniq | wc -l)! Thanks to all of them! Thanos wouldn't be what it is without all of their effort and time! ❤️

Added

• #4394 Add error logs to receiver when write request rejected with invalid replica
• #4403 UI: Add sorting and filtering to flags page
• #4299 Tracing: Add tracing to exemplar APIs.
• #4327 Add environment variable substitution to all YAML configuration flags.
• #4239 Add penalty based deduplication mode for compactor.
• #4292 Receive: Enable exemplars ingestion and querying.
• #4392 Tools: Added --delete-blocks to bucket rewrite tool to mark the original blocks for deletion after rewriting is done.
• #3970 Azure: Adds more configuration options for Azure blob storage. This allows for pipeline and reader specific configuration. Implements HTTP transport configuration options. These options allows for more fine-grained control on timeouts and retries. Implements MSI authentication as second method of authentication via a service principal token.
• #4406 Tools: Add retention command for applying retention policy on the bucket.
• #4430 Compact: Add flag downsample.concurrency to specify the concurrency of downsampling blocks.
• #4231 Receive: Implemented the receive dual-mode. This means that it is now you can run receive as ingester (TSDB only, no hashring awareness), as router (no TSDB, only forwarding mode) and old mode (router + ingester) in one service. Consider splitting functionality if you want to scale your hashring into multiple rings or you want horizontally autoscale your ingester more often.
• #3678 UI: add support for new duration format in graph range input. Now it is possible to use composite durations such as 5m30s.

Fixed

• #4384 Fix the experimental PromQL editor when used on multiple line.
• #4342 ThanosSidecarUnhealthy doesn't fire if the sidecar is never healthy
• #4388 Receive: fix bug in forwarding remote-write requests within the hashring via gRPC when TLS is enabled on the HTTP server but not on the gRPC server.
• #4340 UI: now displays the duration and all annotations of an alert in the alerts page.
• #4348 Fixed parsing of the port in the log middleware.
• #4417 UI: fixed the night mode in Bucket UI.
• #4442 Ruler: fix SIGHUP reload signal not working.

Changed

• #4354 Receive: use the S2 library for decoding Snappy data; saves about 5-7% of CPU time in the Receive component when handling incoming remote write requests
• #4369 Build: Pin upgrade Alpine's version.
• #4404 Receive: added extra validation for the tenant's label name. Some unsupported formats could have passed before.

v0.22.0-rc.0

Highlights

• Added offline deduplication that helps saving even more space on the remote object storage & increases query performance
• The receive dual-mode has been implemented
• Receive now is able to ingest exemplars and it is possible to query them
• Receive now uses a few percent less of CPU time with the same load

This release contains improvements from 27 authors (git log --pretty="%ae" origin/release-0.21...origin/release-0.22 | sort | uniq | wc -l)! Thanks to all of them! Thanos wouldn't be what it is without all of their effort and time! ❤️

Added

• #4394 Add error logs to receiver when write request rejected with invalid replica
• #4403 UI: Add sorting and filtering to flags page
• #4299 Tracing: Add tracing to exemplar APIs.
• #4327 Add environment variable substitution to all YAML configuration flags.
• #4239 Add penalty based deduplication mode for compactor.
• #4292 Receive: Enable exemplars ingestion and querying.
• #4392 Tools: Added --delete-blocks to bucket rewrite tool to mark the original blocks for deletion after rewriting is done.
• #3970 Azure: Adds more configuration options for Azure blob storage. This allows for pipeline and reader specific configuration. Implements HTTP transport configuration options. These options allows for more fine-grained control on timeouts and retries. Implements MSI authentication as second method of authentication via a service principal token.
• #4406 Tools: Add retention command for applying retention policy on the bucket.
• #4430 Compact: Add flag downsample.concurrency to specify the concurrency of downsampling blocks.
• #4231 Receive: Implemented the receive dual-mode. This means that it is now you can run receive as ingester (TSDB only, no hashring awareness), as router (no TSDB, only forwarding mode) and old mode (router + ingester) in one service. Consider splitting functionality if you want to scale your hashring into multiple rings or you want horizontally autoscale your ingester more often.
• #3678 UI: add support for new duration format in graph range input. Now it is possible to use composite durations such as 5m30s.

Fixed

• #4384 Fix the experimental PromQL editor when used on multiple line.
• #4342 ThanosSidecarUnhealthy doesn't fire if the sidecar is never healthy
• #4388 Receive: fix bug in forwarding remote-write requests within the hashring via gRPC when TLS is enabled on the HTTP server but not on the gRPC server.
• #4340 UI: now displays the duration and all annotations of an alert in the alerts page.
• #4348 Fixed parsing of the port in the log middleware.
• #4417 UI: fixed the night mode in Bucket UI.

Changed

• #4354 Receive: use the S2 library for decoding Snappy data; saves about 5-7% of CPU time in the Receive component when handling incoming remote write requests
• #4369 Build: Pin upgrade Alpine's version.
• #4404 Receive: added extra validation for the tenant's label name. Some unsupported formats could have passed before.

v0.21.1

Fixed

• #4308 Sidecar: reloader: fix output config file permission

v0.21.0

Highlights

• Add TLS and basic authentication to Thanos APIs.
• The UI has a dark theme support now.
• Label Matchers for various APIs
• Extra statistics for query API

Added

• #4117 Mixin: new alert ThanosReceiveTrafficBelowThreshold to flag if the ingestion average of the last hour dips below 50% of the ingestion average for the last 12 hours.
• #4107 Store: LabelNames and LabelValues now support label matchers.
• #3940 Sidecar: Added matchers support to LabelValues
• #4171 Docker: Busybox image updated to latest (1.33.1)
• #4175 Added Tag Configuration Support Lightstep Tracing
• #4176 Query API: Adds optional Stats param to return stats for query APIs
• #4125 Rule: Add --alert.relabel-config / --alert.relabel-config-file allowing to specify alert relabel configurations like Prometheus
• #4211 Add TLS and basic authentication to Thanos APIs
• #4249 UI: add dark theme
• #3707 Tools: Added --rewrite.to-relabel-config to bucket rewrite tool to support series relabel from given blocks.

Fixed

• #4105 Tools: Add glob support for filepath in tools command

Changed

• #4223 Query: federated exemplars API only add replica labels to series labels, not to exemplar labels.