do not panic on rebalance during server restarts #19563
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
harshavardhana
force-pushed
the
fix-rebalance
branch
from
2c9991f
to
8c10754
Compare
harshavardhana
changed the title
harshavardhana
force-pushed
the
fix-rebalance
branch
2 times, most recently
from
96eb057
to
568e8a0
Compare
This PR makes a feasible approach to handle all the scenarios that we must face to avoid returning "panic." Instead, we must return "errServerNotInitialized" when a bucketMetadataSys.Get() is called, allowing the caller to retry their operation and wait. Bonus fix the way data-usage-cache stores the object. Instead of storing usage-cache.bin with the bucket as `.minio.sys/buckets`, the `buckets` must be relative to the bucket `.minio.sys` as part of the object name. Otherwise, there is no way to decommission entries at `.minio.sys/buckets` and their final erasure set positions. A bucket must never have a `/` in it. Adds code to read() from existing data-usage.bin upon upgrade.
harshavardhana
force-pushed
the
fix-rebalance
branch
from
568e8a0
to
6d613cc
Compare
krisis
approved these changes
Apr 22, 2024
| stopFn(err) | ||
| if errors.Is(err, errServerNotInitialized) || errors.Is(err, errBucketMetadataNotInitialized) { | ||
| continue |
There was a problem hiding this comment.
Ideally, we should retry this bucket again a few times before we move over to the next bucket. This (fix) is an improvement from what we have now. I can send a PR to tackle this subsequently.
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Community Contribution License
All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request, I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.
Description
do not panic on rebalance during server restarts
Motivation and Context
This PR makes a feasible approach to handle all the scenarios
that we must face to avoid returning "panic."
Instead, we must return "errServerNotInitialized" when a
bucketMetadataSys.Get() is called, allowing the caller to
retry their operation and wait.
Bonus fix the way data-usage-cache stores the object.
Instead of storing usage-cache.bin with the bucket as
.minio.sys/buckets, thebucketsmust be relativeto the bucket
.minio.sysas part of the object name.Otherwise, there is no way to decommission entries at
.minio.sys/bucketsand their final erasure set positions.A bucket must never have a
/in it. Adds code to read()from existing data-usage.bin upon upgrade.
How to test this PR?
CI/CD should test all the scenarios
Types of changes
Checklist:
commit-idorPR #here)