HDDS-10720. Datanode volume DU reserved percent should have a non-zero default value. #6561
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Looks like there's other tests that were depending on 0 being the default value. Let me get them fixed on my fork before running CI here. |
The mocked space usage class does not work it.
|
Green CI on my fork, ready for review. |
adoroszlai
approved these changes
May 3, 2024
|
Thanks for the review @adoroszlai |
jojochuang
pushed a commit
to jojochuang/ozone
that referenced
this pull request
May 29, 2024
…o default value. (apache#6561) (cherry picked from commit 1324e95)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently there are two ways to reserve space in datanode volumes:
hdds.datanode.dir.du.reserved.percentallows specifying a percentage of the volume's space to be unused. It applies to all volumeshdds.datanode.dir.du.reservedallows specifying a map of volume name to bytes reserved. Since it depends on a volume path, it cannot have a default value.By default Ozone should not allow datanode volumes to get 100% full. This can cause the drive to "lock up" because some operations like block delete that would free up space still need extra disk space before they can complete because they must append to the RocksDB WAL. Once encountered, such issues are difficult to resolve. Add a default value for
hdds.datanode.dir.du.reserved.percentto prevent this from happening.A default value of
0.0001fis currently chosen. This is 0.01% which reserves 1GB out of a 10TB volume, 1MB out of a 1TB volume, etc. Ideally we could reserve a fixed size (like 1GB) regardless of drive size, but we would need to re-work the configs before we can do that which might need more discussion. See HDDS-10721.This PR also fixes a few other bugs that prevented tests from passing after the change:
hdds.datanode.dir.du.reserved.percentwould not be used.hdds.datanode.dir.du.reserved. This may have passed in CI but was failing due to my local filesystem setup.What is the link to the Apache JIRA
HDDS-10720
How was this patch tested?
Unit test added.