Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement CREATE FUNCTION for dynamic user-defined function creation #18

Closed
psarna opened this issue Oct 5, 2022 · 0 comments · Fixed by #45
Closed

Implement CREATE FUNCTION for dynamic user-defined function creation #18

psarna opened this issue Oct 5, 2022 · 0 comments · Fixed by #45
Assignees
@psarna

Comments

@psarna
Copy link
Collaborator

psarna commented Oct 5, 2022

In order to provide best developer experience, it should be possible to create and register WebAssembly-based user-defined functions right from the CQL interface, without having to recompile libSQL or register it at the higher code layer, e.g. from a driver.

For that purpose, there already exists a CREATE FUNCTION statement and we should consider adding it to libSQL, with WebAssembly as the supported language. In order to be more or less human-readable, this statement could accept function body in WebAssembly text format (.wat), and then compile it locally, or just in the blob form, in case somebody prefers to avoid the recompilation.

Functions created by CREATE FUNCTION could be stored in a system table, 100% cached in memory, where they can be easily inspected and verified.

Then, the existing code paths for looking up SQL functions can be extended to also peek into the table mentioned above.
This was referenced Oct 5, 2022
Closed
Open
@psarna psarna self-assigned this Oct 7, 2022
This was referenced Oct 7, 2022
Closed
Merged
penberg added a commit that referenced this issue Nov 21, 2022
@penberg
Merge 'Allow registering and executing WebAssembly functions' from Pi…
fd2aae3 
…otr Sarna (#45)

This series implements a mechanism for registering and running Wasm
functions. The current runtime of choice is wasmtime and its
libwasmtime.so library with C bindings (but a switch to Rust should be
considered, because that's the native language of wasmtime and the only
interface which offers all of its features).

It operates on a very crude ABI (ref:#16), where ints and doubles are
passed to/from WebAssembly as is, and for strings/blobs/null it passes a
pointer to a structure:

string: [1 byte for type specification][data]
blob: [1 byte for type specification][4 bytes of size][data]
null: [1 byte for type specification]
The way it's implemented now is twofold:

There's an internal run_wasm function, capable of running WebAssembly
and translating the parameter types from and to the Wasm module A
dynamic lookup table, currently a regular SQL table: CREATE TABLE
libsql_wasm_func_table(name text PRIMARY KEY, body text). The table can
be initialized from C code by calling
libsql_try_initialize_wasm_func_table() or from shell by using a
.init_wasm_func_table command.
After creating and filling the new meta-table, when a function call is
used in a statement, e.g. SELECT id, fib(id) FROM t, and function fib is
neither built-in nor user-defined, it will be looked up in the table. If
found, its body will be assumed to hold valid WebAssembly code, compiled
and run.

In order to enable WebAssembly integration, run configure with
./configure --enable-wasm-runtime parameter.

A few examples WebAssembly-based user-defined functions coded in Rust
can be found here: https://github.com/psarna/libsql_bindgen

Here's an inline demo for testing purposes, with a WebAssembly fibonacci
sequence already compiled from Rust and copied in-place:

```
.init_wasm_func_table

CREATE FUNCTION fib LANGUAGE wasm AS '
(module
 (type (;0;) (func (param i64) (result i64)))
 (func $fib (type 0) (param i64) (result i64)
 (local i64)
 i64.const 0
 local.set 1
 block ;; label = @1
 local.get 0
 i64.const 2
 i64.lt_u
 br_if 0 (;@1;)
 i64.const 0
 local.set 1
 loop ;; label = @2
 local.get 0
 i64.const -1
 i64.add
 call $fib
 local.get 1
 i64.add
 local.set 1
 local.get 0
 i64.const -2
 i64.add
 local.tee 0
 i64.const 1
 i64.gt_u
 br_if 0 (;@2;)
 end
 end
 local.get 0
 local.get 1
 i64.add)
 (memory (;0;) 16)
 (global $__stack_pointer (mut i32) (i32.const 1048576))
 (global (;1;) i32 (i32.const 1048576))
 (global (;2;) i32 (i32.const 1048576))
 (export "memory" (memory 0))
 (export "fib" (func $fib)))
';

CREATE TABLE IF NOT EXISTS example(id int PRIMARY KEY);
INSERT OR REPLACE INTO example(id) VALUES (7);
INSERT OR REPLACE INTO example(id) VALUES (8);
INSERT OR REPLACE INTO example(id) VALUES (9);
SELECT id, fib(id) FROM example;
```

This series also comes with syntactic sugar for registering and
deregistering Wasm functions dynamically via SQL: CREATE FUNCTION and
DROP FUNCTION: Fixes #18

Fixes #17
penberg added a commit to penberg/libsql that referenced this issue Jun 19, 2023
@penberg
Merge pull request tursodatabase#18 from chiselstrike/tableid
1dd92b8 
mvcc: update to the new mvcc-rs API for table_id and row_id
MarinPostma added a commit that referenced this issue Oct 17, 2023
@bors @MarinPostma
Merge #33
15c75e7 
33: batch query r=penberg a=MarinPostma

Add support for batch queries

This was more work than I expected, because of the different protocols we support and different modes of operations. Queries are always batched now (singular queries are in batches on 1).

If any statement in the batch does not parse, then the whole batch is rejected.

fix #18 

Co-authored-by: ad hoc <postma.marin@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@psarna psarna

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow registering and executing WebAssembly functions psarna/libsql
1 participant
@psarna