Add optional ring support to overrides-exporter #3908
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
flxbk
force-pushed
the
felix/exporter-ring
branch
2 times, most recently
from
b6ec9de
to
65ee964
Compare
56quarters
reviewed
Jan 10, 2023
flxbk
force-pushed
the
felix/exporter-ring
branch
2 times, most recently
from
9383723
to
c03b2b6
Compare
flxbk
changed the title
Co-authored-by: Marco Pracucci <marco@pracucci.com>
flxbk
force-pushed
the
felix/exporter-ring
branch
from
e2e8e20
to
cccf038
Compare
pracucci
reviewed
Jan 12, 2023
pracucci
reviewed
Jan 12, 2023
pracucci
approved these changes
Jan 12, 2023
zenador
pushed a commit
that referenced
this pull request
Jan 13, 2023
* Use assert statements instead of handmade checks (#3936) jsonnet already has an 'assert <assertion>' statement, there's no need to craft handmade 'if <assertion> then null else error'. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Fixes templating parsing for manually created vars (#3921) When you create a dashboard variable from within the Grafana UI, it creates the query field as an object. The object contains a refId as well as the original query. The existing code expects the query to be a string, not an object and will therefor miss any metrics used if the variable was created in the UI. * Add optional ring support to overrides-exporter (#3908) * Add optional ring support to overrides-exporter * add docs * add config for read-write mode * fix import * fix naming, swallow error on ring failure * make format * make doc * add license * inline ownership check * add memberlist dependency for overrides exporter * basic integration test * [wip] troubleshoot ring membership * start ring client * fix instance address * Move code * wait on active instance state at startup * fix copy-pasta * improve integration tests * refactor * formatting * inline * move code around * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> * incorporate review feedback * basic unit test * mock ring in unit test * format * clean up * cosmetics/changelog * cosmetic * cosmetic * changelog/log * improve waiting in tests * use GetAddresses() Co-authored-by: Marco Pracucci <marco@pracucci.com> * Update test Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Jack <57678801+mothershipper@users.noreply.github.com> Co-authored-by: Felix Beuke <felix.j.beuke@gmail.com> Co-authored-by: Marco Pracucci <marco@pracucci.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
In read-write deployment mode, every backend replica includes an overrides-exporter that exports per-tenant limit metrics, leading to duplicate metrics being exposed if the number of replicas is greater than one. This PR adds experimental support for a ring to overrides-exporter, which is used to uniquely shard tenant metrics to a given instance and avoid the export of duplicate metrics.
While the solution proposed in this PR works for periods of stable service operations, it is still prone to duplicate exports during rollouts because of the effects of tenant resharding. This problem will be addressed in a follow-up PR.
Which issue(s) this PR fixes or relates to
Fixes #3806
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]