You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am implementing a part of my Python project in Rust. I need to use a tokenizer deep in the Rust code, but I want the Python code to decide what tokenizer to use. I would therefore like to pass a tokenizer from Python to Rust. Since the Python wrapper is supposed to be thin, I have hope that this is possible. If it is not, I'll probably just use Pyo3 to call Python from Rust, but that would both be uglier and feel silly to call Python code that only calls right back into Rust.
What I have tried looks something like this:
use tokenizers::Tokenizer;
#[pyfunction]
pub fn foo(tokenizer: &Tokenizer) {
baz(tokenizer)
}
but Pyo3 gives me the error "the trait bound &tokenizers::Tokenizer: pyo3::PyClass is not satisfied". Looking at the source, this makes sense since it is the PyTokenizer struct in the crate tokenizers-python that wraps Tokenizer and is turned into a Python class. If I could access the PyTokenizer struct in Rust I might therefore be able to fix the problem, but since tokenizers-python is not available on crates.io this is difficult.
Is there some way to pass a tokenizer from Python to Rust?
The text was updated successfully, but these errors were encountered:
SOrry for the late reply here, I think pyo3 is required for this TBH. We have example of this, we use wrappers and match types to switch between model or decoder objects.
I am implementing a part of my Python project in Rust. I need to use a tokenizer deep in the Rust code, but I want the Python code to decide what tokenizer to use. I would therefore like to pass a tokenizer from Python to Rust. Since the Python wrapper is supposed to be thin, I have hope that this is possible. If it is not, I'll probably just use Pyo3 to call Python from Rust, but that would both be uglier and feel silly to call Python code that only calls right back into Rust.
What I have tried looks something like this:
but Pyo3 gives me the error "the trait bound
&tokenizers::Tokenizer: pyo3::PyClass
is not satisfied". Looking at the source, this makes sense since it is thePyTokenizer
struct in the cratetokenizers-python
that wrapsTokenizer
and is turned into a Python class. If I could access thePyTokenizer
struct in Rust I might therefore be able to fix the problem, but sincetokenizers-python
is not available on crates.io this is difficult.Is there some way to pass a tokenizer from Python to Rust?
The text was updated successfully, but these errors were encountered: