Draft release notes for 2.11 #6702

leizor · 2023-11-22T00:18:37Z

What this PR does

This PR adds release notes for Mimir 2.11.0.

Which issue(s) this PR fixes or relates to

#6670

Checklist

Tests updated.
Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
about-versioning.md updated with experimental features.

charleskorn · 2023-11-22T06:09:52Z

docs/sources/mimir/release-notes/v2.11.md

+- The CLI flag `querier.prefer-streaming-chunks-from-ingesters`.
+- The CLI flag `querier.minimize-ingester-requests`.


These flags are still considered experimental - the default value has changed to enable these features by default in this release though.

The plan is to remove the flags entirely in a future release, so these features are just always enabled by default.

Got it, I'll remove them from here.

charleskorn · 2023-11-22T06:10:34Z

docs/sources/mimir/release-notes/v2.11.md

+- **Sampled logging of errors in the ingester.** A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the `-ingester.error-sample-rate` CLI flag.
+- **Add total request size instance limit for ingesters.** This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the `-ingester.instance-limits.max-inflight-push-requests-bytes` CLI flag.
+- **Reduce the resolution of incoming native histograms samples** if the incoming sample has too many buckets compared to `-validation.max-native-histogram-buckets`. This is enabled by default but can be turned off by setting the `-validation.reduce-native-histogram-over-max-buckets` CLI flag to `false`.
+- **Include a `Retry-After` header in recoverable error responses from the distributor.** This can protect your Mimir cluster from clients that default to retrying very quickly. Enable this feature by setting the `-distributor.retry-after-header.enabled` CLI flag.


Is it worth mentioning some examples of clients that respect this header?

docs/sources/mimir/release-notes/v2.11.md

colega · 2023-11-22T10:08:30Z

docs/sources/mimir/release-notes/v2.11.md

+
+In Grafana Mimir 2.11 the following behavior has changed:
+
+- The distributor `Push()` endpoint will now return the following gRPC codes instead of HTTP status codes:


As a user, I don't know what is the distributor's Push() endpoint, is that where I send my HTTP requests?

I think I was unintentionally a little vague because I wasn't 100% sure, but this is the distributor's gRPC endpoint, right?

Hum, I think it's only misleading here. Distributor doesn't have a documented gRPC API as far as I can see. This change is only internal, as this endpoint is called from the http wrapper, and these errors are translated to same as previously.

I would just remove this section.

cc @duricanikolic for more context.

Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>

pstibrany · 2023-11-23T13:59:29Z

docs/sources/mimir/release-notes/v2.11.md

+## Features and enhancements
+
+- **Sampled logging of errors in the ingester.** A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the `-ingester.error-sample-rate` CLI flag.
+- **Add total request size instance limit for ingesters.** This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the `-ingester.instance-limits.max-inflight-push-requests-bytes` CLI flag.


To make this work efficiently (without reading request to memory first), this needs to be used with -ingester.limit-inflight-requests-using-grpc-method-limiter.

Got it, thanks!

pstibrany · 2023-11-23T14:02:17Z

docs/sources/mimir/release-notes/v2.11.md

+- The distributor gRPC push endpoint will now return the following gRPC codes instead of HTTP status codes:
+  - 202 (accepted) code is replaced with 6 (`ALREADY_EXISTS`).
+  - 400 (bad request) code is replaced with 9 (`FAILED_PRECONDITION`).
+  - 429 (too many requests) and the non-standard 529 (service is overloaded) codes are replaced with 8 (`RESOURCE_EXHAUSTED`).


Distributor gRPC push endpoints are not part of public API. I don't think this needs to be mentioned in release notes.

dimitarvdimitrov

thanks for writing this. I see you've gone through a lot of PRs to get this done 😅

docs/sources/mimir/release-notes/v2.11.md

dimitarvdimitrov · 2023-11-27T19:49:05Z

docs/sources/mimir/release-notes/v2.11.md

+- **Reduce the resolution of incoming native histograms samples** if the incoming sample has too many buckets compared to `-validation.max-native-histogram-buckets`. This is enabled by default but can be turned off by setting the `-validation.reduce-native-histogram-over-max-buckets` CLI flag to `false`.
+- **Include a `Retry-After` header in recoverable error responses from the distributor.** This can protect your Mimir cluster from clients including Prometheus that default to retrying very quickly. Enable this feature by setting the `-distributor.retry-after-header.enabled` CLI flag.
+- **Improved query-scheduler performance under load.** This is particularly apparent for clusters with large numbers of queriers.
+- **Ingester to querier chunks streaming** reduces the memory utilization of queriers and reduces the likelihood of OOMs.


is this a new feature or was it just enabled by default now?

It's enabled by default now.

we can mention it if you want

docs/sources/mimir/release-notes/v2.11.md

Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

dimitarvdimitrov

LGTM, thanks for writing this!

leizor · 2023-11-28T19:36:39Z

Thanks for all the input, everybody!

* Draft release notes for 2.11 * Apply suggestions from code review Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com> * More code review responses * More mode code review responses * Update docs/sources/mimir/release-notes/v2.11.md Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * More more more code review responses --------- Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com> Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

leizor marked this pull request as ready for review November 22, 2023 00:23

leizor requested review from a team as code owners November 22, 2023 00:23

Draft release notes for 2.11

48dd661

leizor force-pushed the leizor/2.11-release-notes branch from 7cf37d1 to 48dd661 Compare November 22, 2023 00:34

charleskorn reviewed Nov 22, 2023

View reviewed changes

colega reviewed Nov 22, 2023

View reviewed changes

leizor and others added 2 commits November 22, 2023 10:10

Apply suggestions from code review

1c61c6c

Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>

More code review responses

fe750e8

pstibrany reviewed Nov 23, 2023

View reviewed changes

More mode code review responses

4be83b1

dimitarvdimitrov reviewed Nov 27, 2023

View reviewed changes

leizor and others added 2 commits November 27, 2023 16:00

Update docs/sources/mimir/release-notes/v2.11.md

56da2c4

Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

More more more code review responses

a34fb24

leizor force-pushed the leizor/2.11-release-notes branch from 25548bd to a34fb24 Compare November 28, 2023 02:48

dimitarvdimitrov approved these changes Nov 28, 2023

View reviewed changes

leizor merged commit 0da3ce0 into main Nov 28, 2023
28 checks passed

leizor deleted the leizor/2.11-release-notes branch November 28, 2023 19:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft release notes for 2.11 #6702

Draft release notes for 2.11 #6702

leizor commented Nov 22, 2023 •

edited

charleskorn Nov 22, 2023

leizor Nov 22, 2023

charleskorn Nov 22, 2023

colega Nov 22, 2023

leizor Nov 22, 2023

colega Nov 23, 2023

pstibrany Nov 23, 2023

leizor Nov 24, 2023

pstibrany Nov 23, 2023

dimitarvdimitrov left a comment

dimitarvdimitrov Nov 27, 2023

charleskorn Nov 27, 2023

dimitarvdimitrov Nov 28, 2023

dimitarvdimitrov left a comment

leizor commented Nov 28, 2023

		- The CLI flag `querier.prefer-streaming-chunks-from-ingesters`.
		- The CLI flag `querier.minimize-ingester-requests`.


		In Grafana Mimir 2.11 the following behavior has changed:

		- The distributor `Push()` endpoint will now return the following gRPC codes instead of HTTP status codes:

Draft release notes for 2.11 #6702

Draft release notes for 2.11 #6702

Conversation

leizor commented Nov 22, 2023 • edited

What this PR does

Which issue(s) this PR fixes or relates to

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimitarvdimitrov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimitarvdimitrov left a comment

Choose a reason for hiding this comment

leizor commented Nov 28, 2023

leizor commented Nov 22, 2023 •

edited