-
Notifications
You must be signed in to change notification settings - Fork 298
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade rustpython #3063
Comments
@killme2008 @discord9 I remember that in an offline discussion, it's thrown that we might finally use PyO3 only. Is there any concrete reason and plan? IIRC we initially add RustPython to decouple the dependency to system python interpreter. But now we think it's a fair requirement? |
The main reason is that RustPython is far from production currently, and Python and PyO3 are developing rapidly. |
@killme2008 can we first dropping the code related to RustPython and recommend to use the PyO3 backend version? For PyO3 backend version, the major challenge is supported range of Python version. @messense do you have some inputs in this topic? We're a binary rather than a lib, so we may not release multiple artifacts to cover the py versions like OpenDAL. Instead, we're trying to use abi3 to expand the supported versions range. But we still lack knowledge in how to move forward in this direction. |
Embedding python interpreter is an underexplored area in pyo3,
Will this work for greptimedb? |
@messense Thanks for your information. Let me think of it ... Cross-ref: https://pyo3.rs/v0.21.0/building-and-distribution |
@messense we may not use greptimedb/src/script/Cargo.toml Line 57 in d5a948a
But yet to test against multiple Python version. |
Found that we currently depend on RustPython's AST parsing function to extra metadata from decorators: @coprocessor(args=[], returns=["value"], backend="pyo3")
def test_numpy() -> vector[i64]:
try:
import numpy as np
import pyarrow as pa
except ImportError as e:
# Python didn't have numpy or pyarrow
print("Warning: no pyarrow or numpy found in current python", e)
return vector([0, 1, 2, 3, 4, 5, 6, 7, 8, 9,])
from greptime import vector
v = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9,])
v = pa.array(v)
v = vector.from_pyarrow(v)
v = vector.from_numpy(v.numpy())
v = v.to_pyarrow()
v = v.to_numpy()
v = vector.from_numpy(v)
return v |
RustPython's parser is now a standalone crate, which is battle-tested in Real Python Usecase like Python Lint |
No. The API of RustPython has changed a lot, ref - RustPython/Parser#119. I generally doubt if such source edit approach the way to go. Maybe a Python UDF like PL/Python or a Python worker that can connect to the SQL server a good alternative. |
https://github.com/astral-sh/ruff/blob/f779babc5f725ccfe9c4e08b26d1abe54166619d/CONTRIBUTING.md?plain=1#L641 |
Supersedes by #3777. |
What type of enhancement is this?
Tech debt reduction
What does the enhancement do?
Our rustpython version looks too old, maybe we need to upgrade it.
Implementation challenges
No response
The text was updated successfully, but these errors were encountered: