feat(udf): support embedded python udf

[wangrunji0408]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This PR allows user to define Python UDFs that will be run in an embedded Python interpreter in RisingWave.

# scalar function
dev=> create function gcd(a int, b int) returns int language python as $$
def gcd(a, b):
    while b != 0:
        a, b = b, a % b
    return a
$$;
CREATE_FUNCTION
dev=> select gcd(15, 25);
 gcd 
-----
   5
(1 row)

# table function
dev=> create function series(n int) returns table (x int) language python as $$
def series(n):
    for i in range(n):
        yield i
$$;
CREATE_FUNCTION
dev=> select series(5);
 series 
--------
      0
      1
      2
      3
      4
(5 rows)

This is very similar to #14513. We use pyo3 to embed a Python library into RisingWave. The minimum supported Python version is 3.12. Because we require sub-interpreters and per-interpreter GIL to run different UDFs in parallel.

For safety reasons, functions are limited to pure computational logic. We only allow importing packages in a whitelist and have removed some builtin functions such as exit and open. The goal is to create a sandbox for untrusted code. However, it is not absolute safe, at least for now. It seems that Python interpreter doesn't provide a way to limit CPU or memory usage. The system can be easily blocked by an infinite loop in UDF. Until this problem is resolved, it is be better to only open this feature to admin users.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

Support embedded Python UDFs. Functions will run inside RisingWave and don't require external UDF servers.

Documentations will be updated later.

[fuyufjh]

Would this Python interpreter be able to execute any command like os.system(command)?

[wangrunji0408]

Would this Python interpreter be able to execute any command like os.system(command)?

No. We only allow importing packages in a whitelist and have removed dangerous functions such as exit and open, before running untrusted code from user.

https://github.com/risingwavelabs/arrow-udf/blob/273c8c50ddca10a8988691e8d95374f3af371548/arrow-udf-python/src/lib.rs#L58

[BugenZhao]

Our embedded UDF is getting more and more exciting. 🤩

BTW, may I ask about the status of Python WASM ecosystem? I suppose it's not that functional so that we chose to run it on a Python interpreter and handle the isolation and security ourselves.

[wangrunji0408]

BTW, may I ask about the status of Python WASM ecosystem? I suppose it's not that functional so that we chose to run it on a Python interpreter and handle the isolation and security ourselves.

We have been able to run Python UDFs on WASM using a prebuilt Python WASM image provided by vmware labs. The ecosystem looks fine and WASM does provide better isolation. However, my biggest concern is performance. According to our benchmark result, Python is already very slow (100x slower than native code), running it on WASM makes it 10x slower. I think the performance impact is far more than security benefits, making it less useful in practice. So I don't choose WASM for Python UDF. 😢

[fuyufjh]

This PR LGTM! Will take a look at https://github.com/risingwavelabs/arrow-udf/tree/main/arrow-udf-python later