Prometheus: stream HTTP responses (a double digit % reduction of both CPU and memory footprint) (backport #14885) #15018
+21
−55
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`prometheus_text_format:format/1` produces a binary of the format for the entire registry. For clusters with many resources, this can lead to large replies from `/metrics/[:registry]` especially for large registries like `per-object`. Instead of formatting the response and then sending it, we can stream the response by taking advantage of the new `format_into/3` callback (which needs to be added upstream to the `prometheus` dep). This uses `cowboy_req:stream_body/3` to stream the iodata as `prometheus` works through the registry. This should hopefully be a nice memory improvement. The other benefit is that results are sent eagerly. For a stress-testing example, 1. `make run-broker` 2. `rabbitmqctl import_definitions path/to/100k-classic-queues.json` 3. `curl -s localhost:15692/metrics/per-object` Before this change `curl` would wait for around 8 seconds and then the entire response would arrive. With this change the results start streaming in immediately. (cherry picked from commit 93a014c)
The Cowboy stream handler `cowboy_compress_h` is already enabled for this endpoint so there's no need to perform gzipping manually. This change deletes the custom handling of accept-encoding. We previously returned a content-encoding of "identity" for non-gzipped requests, but according to <https://www.rfc-editor.org/rfc/rfc9110.html#section-8.4>, the content-encoding response header should not be set to identity. The test case is accurate anyways: the response is text, not something compressed. (cherry picked from commit eca7881)
Collaborator
|
@Mergifyio backport v4.1.x |
Author
✅ Backports have been created
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
prometheus_text_format:format/1produces a binary of the format for the entire registry. For clusters with many resources, this can lead to large replies from/metrics/[:registry]especially for large registries likeper-object. Instead of formatting the response and then sending it, we can stream the response by taking advantage of the newformat_into/3callback (which needs to be added upstream to theprometheusdep). This usescowboy_req:stream_body/3to stream the iodata asprometheusworks through the registry.This should hopefully be a nice memory improvement. The other benefit is that results are sent eagerly. For a stress-testing example,
make run-brokerrabbitmqctl import_definitions path/to/100k-classic-queues.jsoncurl -s localhost:15692/metrics/per-objectBefore this change
curlwould wait for around 8 seconds and then the entire response would arrive. With this change the results start streaming in immediately.Discussed in #14865
This is an automatic backport of pull request #14885 done by Mergify.