bottomless2: perform store job #1405
Conversation
MarinPostma
commented
May 22, 2024
MarinPostma
commented
- implement scheduler
- move bottomless2 to libsql-wal
- add async fs methods
- implement perform job
MarinPostma
force-pushed
the
perform-bottomless-job
branch
3 times, most recently
from
58dce40
to
71283a2
Compare
.cargo/config.toml
Outdated
There was a problem hiding this comment.
just quieting warnings
MarinPostma
force-pushed
the
perform-bottomless-job
branch
from
71283a2
to
ec0ea1d
Compare
libsql-wal/Cargo.toml
Outdated
| @@ -19,3 +19,12 @@ thiserror = "1.0.58" | |||
| tracing = "0.1.40" | |||
| walkdir = "2.5.0" | |||
| zerocopy = { version = "0.7.32", features = ["derive", "alloc"] } | |||
| tokio = { version = "1", features = ["full"] } | |||
There was a problem hiding this comment.
do we need all the features?
There was a problem hiding this comment.
Probably not, but we're compiling them anyway. I'll trim later.
libsql-wal/Cargo.toml
Outdated
| @@ -19,3 +19,12 @@ thiserror = "1.0.58" | |||
| tracing = "0.1.40" | |||
| walkdir = "2.5.0" | |||
| zerocopy = { version = "0.7.32", features = ["derive", "alloc"] } | |||
| tokio = { version = "1", features = ["full"] } | |||
| tempfile = "3.10.1" | |||
There was a problem hiding this comment.
nope, it's being used
libsql-wal/src/io/file.rs
Outdated
| let mut written = 0; | ||
|
|
||
| while written != buf.len() { | ||
| written += nix::sys::uio::pwrite(self, &buf[written..], offset as _)?; |
There was a problem hiding this comment.
dont you need to write at offset + written?
There was a problem hiding this comment.
hmm yeah, I need to write a test that catches that
There was a problem hiding this comment.
I was unable to write a test to verify that, no matter how big the payload, it's writing everything in one go...
| // todo: create an async counterpart | ||
| fn tempfile(&self) -> io::Result<Self::TempFile>; | ||
| fn now(&self) -> DateTime<Utc>; | ||
| fn uuid(&self) -> Uuid; |
There was a problem hiding this comment.
uuid is part of the io system 🤣
There was a problem hiding this comment.
it's randomness
There was a problem hiding this comment.
randomness is an input
There was a problem hiding this comment.
actually I'll probably just expose a range 🤔
MarinPostma
force-pushed
the
perform-bottomless-job
branch
from
ec0ea1d
to
c795e92
Compare
MarinPostma
force-pushed
the
perform-bottomless-job
branch
2 times, most recently
from
0a364f0
to
4b47e80
Compare
MarinPostma
force-pushed
the
perform-bottomless-job
branch
from
4b47e80
to
e808398
Compare
| while let Some((page_no_bytes, offset)) = pages.next() { | ||
| let page_no = u32::from_be_bytes(page_no_bytes.try_into().unwrap()); | ||
| let (b, ret) = segment.read_frame_offset_async(offset as _, buffer).await; | ||
| ret.unwrap(); |
There was a problem hiding this comment.
whats this unwrap for?
There was a problem hiding this comment.
Uh yeah, I should return an io error here, I'll do in a follow up
| } | ||
|
|
||
| fn uuid(&self) -> Uuid { | ||
| dbg!(); |
There was a problem hiding this comment.
| dbg!(); |
| @@ -43,7 +52,7 @@ impl<FS: FileSystem> WalManager for LibsqlWalManager<FS> { | |||
| .registry | |||
| .clone() | |||
| .open(db_path.as_ref()) | |||
| .map_err(Into::into)?; | |||
| .map_err(|e| dbg!(e.into()))?; | |||
