[Backports stable/1.3] Preallocate segment files #9778
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a method to preallocate a new file of the expected length, effectively reserving the disk space for later use. The current implementation is a "dumb" one, which simply writes 4KB blocks of 0s until the expected length is reached (meaning the size may be up to 4KB - 1 byte greater than the desired length, which is acceptable for our use case). While potentially slow, this is not in a hot path, and will be later optimized with a native syscall to fallocate.
Adds basic tests for `FileUtil#preallocate`. A new dependency was added, `jnr-posix`, which allows us to check the actual size on disk of the file in UNIX systems. This gives us the accurate number of blocks reserved for the file, on disk, even on file systems with compression or sparse files (i.e. most modern Linux systems), and also ensures that we don't only read the metadata (e.g. what `Files.size` returns) but really guarantee we reserved the disk space.
Allows preallocating segment files if configured to do so. On segment creation, we now preallocate the segment file by default before using it. If it already existed (but was unused by the log so far), we take the easy way out and simply delete/recreate it instead of attempting to grow/shrink it.
Add a new dependency on `jnr-posix` to accurately get the size of the file on disk. This is necessary to avoid tests passing if we only change the file's metadata - for example, when mmap-ing a file with a mapping of length X, then the file will report it has a length of X, even though it has possibly only one block allocated on disk.
|
I essentially just backported #9777 here, so if you review one of the two, they're pretty much the same. |
|
bors merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Backports #9731 to 1.3.x.
Related issues
closes #6504
closes #8099
Definition of Done
Not all items need to be done depending on the issue and the pull request.
Code changes:
backport stable/1.3) to the PR, in case that fails you need to create backports manually.Testing:
Documentation:
Please refer to our review guidelines.