Delete the index for empty tenants #2678
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zalegrala
force-pushed
the
tempoIndexDeletion
branch
from
56301a1
to
0d2f834
Compare
zalegrala
requested review from
joe-elliott,
annanay25,
mdisibio,
mapno,
kvrhdn,
electron0zero,
ie-pham and
stoewer
as code owners
mdisibio
approved these changes
Jul 20, 2023
There was a problem hiding this comment.
Nice cleanup. I think all of the real object stores will "delete" the tenant folder when the last item is removed (i.e. index.json.gz). Will this be a problem for the local backend? It might be used in some cases for non-production setups. The local-blocks processor and the ingester also use a local backend but not with polling so they should be ok.
|
The |
zalegrala
added a commit
to zalegrala/tempo
that referenced
this pull request
Apr 10, 2024
The tenant index deletion was originally put in as TCO win, but did not have the desired effect and surfaced other issues in the system. Related grafana#2678 Related grafana#2754 Related grafana#2781 Related grafana#2878 Related grafana#3115 Related grafana#3223 Due to the number of issues here, and causing considerable noise on the pager, perhaps the right thing to do is back out the tenant deletion. Raising here for discussion.
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
When a tenant stops writing to tempo, the blocks eventually get cleaned up by the retention, but the tenant index is still left and so the tenant is still left in the list when the poller determines which indexes to pull. This can create additional load for no benefit long term if the tenant doesn't return with traffic.
Here we call the recently implemented
Deleteon theRawWriterto remove the tenant indexes from the backend to complete the deletion of all tenant objects when no blocks or compacted blocks have been found to exist.Which issue(s) this PR fixes:
Fixes #2520
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]