Refactor writer process to support direct disk writing #126
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This closes #10, #115, and #124:
We also significantly improve the unit testing of the writer process.
Test plan
Test plan, multiple screenshots
Aligned file, Linux
Unaligned file, Linux
Aligned compressed file, Linux
Aligned file, MacOS
Unaligned file, MacOS
Performance impact?
As it turns out, negligible!
The following tests are run on the
main
branch, with uncompressed 1GB files.Naively running Caligula on the base branch on Linux seems to indicate that this PR causes performance degradation.
However, that's because we aren't properly syncing the file, since
std::io::File::flush
is a no-op. Actually adding in a.sync_data()
at the end of the write indicates that the disk is technically not fully written by that point, so the actual write speed is slower than it looks from the program's end.These speeds are more in line with what we saw above.