(2/3) Commitlog: Add I/O based on regular files #920
Conversation
| /// visible to the iterator. Upon encountering [`io::ErrorKind::UnexpectedEof`], | ||
| /// however, a new iterator should be created using [`Self::commits_from`] | ||
| /// with the last transaction offset yielded. | ||
| pub fn commits(&self) -> impl Iterator<Item = Result<Commit, error::Traversal>> { |
There was a problem hiding this comment.
slash-slash-todo: We're skipping bad commits in case the next commit is good. It's not obvious what to do when the iterator is exhausted and the last commit is bad -- there could be more coming at a later point in time. So for now, we'll yield the error.
This is fine in tooling, but requires special-casing when replaying the log to fold it into a datastore. Maybe a named iterator which can be turned into one which just ignores the error if it's the last item could be handy.
There was a problem hiding this comment.
I settled on yielding the error in iterators, but ignore it in folds (because the consumer of a fold doesn't have access to the traversal itself).
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
802f385
to
c518f99
Compare
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
c518f99
to
67c1884
Compare
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
67c1884
to
f22752a
Compare
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
f22752a
to
f1cd904
Compare
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
f1cd904
to
9e13cd5
Compare
bfops
added
release-any
| use super::Repo; | ||
|
|
||
| const SEGMENT_FILE_EXT: &str = ".stdb.log"; | ||
|
|
There was a problem hiding this comment.
.log extension could be confused easily as being text-based, how about .log.stdb or .stdb.wbl (write behind log)
There was a problem hiding this comment.
This file extension was requested by @Centril. If we don’t like it, an amendment to the spec shall be proposed.
There was a problem hiding this comment.
.log is pretty standard. I think it's the same for kafka IIRC.
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
9e13cd5
to
d4e4dc5
Compare
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
d4e4dc5
to
a7f8dfd
Compare
kim
force-pushed
the
kim/commitlog2-sansio
branch
from
ecdb130
to
03a645e
Compare
kim
force-pushed
the
kim/commitlog2-io
branch
2 times, most recently
from
5f6b3e3
to
f2e3f7d
Compare
Provides a commitlog backing store based on files, and defines the exported `Commitlog` type which fixes the store to the file-based one.
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Provides a commitlog backing store based on files, and defines the exported
Commitlogtype which fixes the store to the file-based one.NOTE: This PR is stacked on top of #919
Expected complexity level and risk
1