-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bottomless2: perform store job #1405
Conversation
MarinPostma
commented
May 22, 2024
- implement scheduler
- move bottomless2 to libsql-wal
- add async fs methods
- implement perform job
58dce40
to
71283a2
Compare
.cargo/config.toml
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just quieting warnings
71283a2
to
ec0ea1d
Compare
libsql-wal/Cargo.toml
Outdated
@@ -19,3 +19,12 @@ thiserror = "1.0.58" | |||
tracing = "0.1.40" | |||
walkdir = "2.5.0" | |||
zerocopy = { version = "0.7.32", features = ["derive", "alloc"] } | |||
tokio = { version = "1", features = ["full"] } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need all the features?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably not, but we're compiling them anyway. I'll trim later.
libsql-wal/Cargo.toml
Outdated
@@ -19,3 +19,12 @@ thiserror = "1.0.58" | |||
tracing = "0.1.40" | |||
walkdir = "2.5.0" | |||
zerocopy = { version = "0.7.32", features = ["derive", "alloc"] } | |||
tokio = { version = "1", features = ["full"] } | |||
tempfile = "3.10.1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dev dep?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nope, it's being used
libsql-wal/src/io/file.rs
Outdated
let mut written = 0; | ||
|
||
while written != buf.len() { | ||
written += nix::sys::uio::pwrite(self, &buf[written..], offset as _)?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dont you need to write at offset + written?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm yeah, I need to write a test that catches that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was unable to write a test to verify that, no matter how big the payload, it's writing everything in one go...
// todo: create an async counterpart | ||
fn tempfile(&self) -> io::Result<Self::TempFile>; | ||
fn now(&self) -> DateTime<Utc>; | ||
fn uuid(&self) -> Uuid; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uuid is part of the io system 🤣
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's randomness
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
randomness is an input
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually I'll probably just expose a range 🤔
ec0ea1d
to
c795e92
Compare
0a364f0
to
4b47e80
Compare
4b47e80
to
e808398
Compare
while let Some((page_no_bytes, offset)) = pages.next() { | ||
let page_no = u32::from_be_bytes(page_no_bytes.try_into().unwrap()); | ||
let (b, ret) = segment.read_frame_offset_async(offset as _, buffer).await; | ||
ret.unwrap(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
whats this unwrap for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Uh yeah, I should return an io error here, I'll do in a follow up
} | ||
|
||
fn uuid(&self) -> Uuid { | ||
dbg!(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dbg!(); |
@@ -43,7 +52,7 @@ impl<FS: FileSystem> WalManager for LibsqlWalManager<FS> { | |||
.registry | |||
.clone() | |||
.open(db_path.as_ref()) | |||
.map_err(Into::into)?; | |||
.map_err(|e| dbg!(e.into()))?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can remove this?