Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support compiling libSQL to Wasm with virtual WAL #695

Merged
merged 10 commits into from Nov 26, 2023

Conversation

psarna
Copy link
Collaborator

@psarna psarna commented Nov 24, 2023

This PR contains a demo which uses Wasmtime runtime to run libSQL in a WebAssembly instance, compiled in WAL mode. Right now it's full of stubs, and a VFS that calls back to the host for each operation. Once these stubs are implemented and we also provide a virtual WAL implementation that calls back to the host, we'll have a solid foundation for running libSQL with replication as a Wasm instance.

Step 1: compile libsql.wasm (should be automated with build.rs, I'm just lazy)

cd libsql-sqlite3
make wasi

Step 2: run the demo

cd ext/libsql-wasi-demo
RUST_LOG=info,libsql_wasi_demo=trace cargo run

Step 3: profit

$ file /tmp/wasm-demo.db*
/tmp/wasm-demo.db:     SQLite 3.x database, last written using SQLite version 3044000, writer version 2, read version 2, file counter 1, database pages 1, cookie 0, schema 0, unknown 0 encoding, version-valid-for 1
/tmp/wasm-demo.db-wal: empty
$ sqlite3 /tmp/wasm-demo.db "pragma journal_mode"
wal

Virtual WAL is not implemented in this patch, but this demo already successfully used WAL mode from a WebAssembly instance and created a WAL.

Next steps (arbitrary order):

  1. forge this demo into a library we can later use to run libsql Wasm instances for users
  2. add virtual WAL support

@MarinPostma
Copy link
Collaborator

@psarna could we base that on WAL v2 😄 It'll never get merged if we keep adding stuff on top of v1 🙏

@MarinPostma
Copy link
Collaborator

ah wait it's actually only VFS

@psarna
Copy link
Collaborator Author

psarna commented Nov 24, 2023

it's only a VFS stub for now, next week I'll fill all the implementation and then add virtual WAL too. But even regular WAL should work if the VFS is properly implemented, and that will be a major win: we'll have proof that libsql runs in WAL mode when compiled to Wasm.

@MarinPostma
Copy link
Collaborator

MarinPostma commented Nov 24, 2023

my point was more that I will need to port this code with wal v2, which means wal v2 is not getting merged any time soon 😢

@psarna
Copy link
Collaborator Author

psarna commented Nov 24, 2023

Why? It isn't in any way intrusive with WAL, I'll just first implement it with what we have, and then port it to walv2 later, but it should be quite trivial (and nonblocking, worst case it breaks for a short while)

It's far from working, but it already properly calls back
from a WebAssembly instance to the host for VFS operations.
We also want the same for virtual WAL and we'll have a solid foundation
for running libSQL in a Wasm runtime.
@psarna
Copy link
Collaborator Author

psarna commented Nov 26, 2023

Breakthrough: the demo managed to use libsql compiled to Wasm in WAL mode, and store actual data in WAL. Next steps - make it a library, and add virtual WAL support.

@psarna psarna marked this pull request as ready for review November 26, 2023 13:22
@psarna psarna added this pull request to the merge queue Nov 26, 2023
Merged via the queue into tursodatabase:main with commit c24c91c Nov 26, 2023
14 of 15 checks passed
@psarna psarna deleted the wasistub branch November 26, 2023 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants