Introduce file system and implement streaming encoding #91
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@tabokie Please fill the description, it is not friendly for others want to know more about this PR with no full description. |
tabokie
commented
Aug 28, 2021
| self.buf.encode_u64(0)?; | ||
| self.buf.encode_u64(0)?; | ||
| } else if self.buf_state != BufState::EntriesFilled { | ||
| // return error |
tabokie
commented
Aug 28, 2021
| enum BufState { | ||
| Uninitialized, | ||
| EntriesFilled, | ||
| Sealed(usize), |
There was a problem hiding this comment.
Commenting the meaning for this counter.
tabokie
commented
Aug 28, 2021
| ftruncate(self.0, offset as i64).map_err(|e| from_nix_error(e, "ftruncate")) | ||
| } | ||
|
|
||
| pub fn allocate(&self, offset: usize, size: usize) -> IoResult<()> { |
There was a problem hiding this comment.
Test that an allocated file reads out 0.
This was referenced Aug 31, 2021
tabokie
added a commit
that referenced
this pull request
Sep 7, 2021
Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie
added a commit
to tabokie/raft-engine
that referenced
this pull request
Sep 8, 2021
Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie
added a commit
to tabokie/raft-engine
that referenced
this pull request
Sep 8, 2021
Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie
added a commit
that referenced
this pull request
Sep 8, 2021
1. Append compressed entries to log batch buffer instead of replacing the buffer. 2. Avoid allocate twice when creating `protobuf::Message` from decompressed slice 3. Use entry reference for log batch append Bench results: ``` Patched: test log_batch::tests::bench_log_batch_add_entry_and_encode ... bench: 136,299 ns/iter (+/- 44,912) test log_batch::tests::bench_log_batch_add_entry_and_encode ... bench: 131,405 ns/iter (+/- 82,630) Master: test log_batch::tests::bench_log_batch_add_entry_and_encode ... bench: 199,314 ns/iter (+/- 95,954) test log_batch::tests::bench_log_batch_add_entry_and_encode ... bench: 197,948 ns/iter (+/- 16,842) Without streaming encoding (#91): test log_batch::tests::bench_log_batch_add_entry_and_encode ... bench: 407,738 ns/iter (+/- 142,417) test log_batch::tests::bench_log_batch_add_entry_and_encode ... bench: 411,079 ns/iter (+/- 111,409) ``` Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie
added a commit
that referenced
this pull request
Sep 9, 2021
Replace file system (#91) with an Allocator-like type `FileBuilder`. File builder is better, it's strongly typed with our own file writer/reader, so that 1) we don't need to propagate `Send` and `Sync` trait bound outside of this library, 2) user writers or readers are promised to use our writer/reader as a backend. There still are some reasonable requirements on user types, such as no buffering and no length altering, which are documented in trait definition. Signed-off-by: tabokie <xy.tao@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: tabokie xy.tao@outlook.com
This PR rearranges internal data access around a standard reader/writer. This way external IO logic can be embedded through the new interface
FileSystem, which allows for building wrapper around said reader/writer.This interface is not ideal as it doesn't ensure implementation to actually leverage the raw IO object. More work needs to be done to factor out code dependence on
LogFdand severFileSystemas a standalone abstraction.This PR also implements streaming encoding.
LogBatchnow encodes user data immediately instead of keeping a copy and encode before actual writing.Stress results: