[v23.1.x] Backport of PR #9430 - Add new metering API to track Kafka traffic #9666
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
graphcareful
changed the title
graphcareful
force-pushed
the
backport-9430-v23.1.x-410
branch
from
481d07f
to
b059ae1
Compare
(cherry picked from commit afc5052)
- These two configuration properties control the bucketing interval and length of history that will be kept for the new usage tracking mechanism. (cherry picked from commit c57926c)
- Fixes: redpanda-data#9076 (cherry picked from commit 80e9d11)
graphcareful
force-pushed
the
backport-9430-v23.1.x-410
branch
from
b059ae1
to
11e8231
Compare
(cherry picked from commit 491f363)
(cherry picked from commit 3339ab0)
(cherry picked from commit 942d64c)
(cherry picked from commit e22d9f0)
(cherry picked from commit e0b6c7c)
- Usage metrics are collected on a per request basis and are added to the collector (kafka::usage_manager) after the request has been fully processed. - This occurs during the response handling phase, two exceptions are accounted for which is traffic on internal topics in produce/fetch requests, this number of bytes is decremented from the total request/response size and forwarded to the usage_manager for recording. (cherry picked from commit b7e2861)
(cherry picked from commit 3aa7da7)
- Return all cached usage results upon GET request to v1/usage (cherry picked from commit f7265cf)
(cherry picked from commit 3388798)
graphcareful
force-pushed
the
backport-9430-v23.1.x-410
branch
from
11e8231
to
0e3bf10
Compare
dlex
reviewed
Mar 27, 2023
dlex
approved these changes
Mar 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover Letter
Backport of PR: #9077
UX Changes
enable_usage: To turn feature on/off (defaulted to off)usage_num_windows: Number of historical windows of data of usage to trackusage_window_width_interval_sec: The length of time in seconds of a historical windowusage_disk_persistance_interval_sec: Length of time between writes of the historical data to disk, for persistence in case of shutdown/crashesRelease Notes
Improvements