Python binding

[mh-northlander]

Create sudachipy conpatible python binding

[mh-northlander]

considering to use PyO3, following hugging-face tokenizers

[eiennohito]

What about Python API FFI like this? Main Idea - hide heavyweight API behind traits and use them via objects from Python side.

pub fn dictionary_from_config(cfg: Config) -> SudachiResult<Arc<dyn PyDictionary>> {
    todo!()
}

pub trait PyDictionary {
    fn tokenizer(&self) -> Box<dyn mut PyTokenize>;
}

pub trait PyTokenise {    
    fn tokenize(&mut self, input: &str, mode: Mode, debug: bool) -> SudachiResult<Vec<Morpheme>>;
}

Morpheme should be owning for Python, internal one would not work, binding should handle copying from Rust types to Python ones. Result also may be non-compatible.

[mh-northlander]

PyO3 can only convert struct, so I'm considering to expose wrapper classes like following to hide rust API:

pub struct PyDictionary {
    inner: Arc<JapaneseDictionary>
}

impl PyDictionary {
    fn from_config(config: Config) -> Self {
        todo!();
    }

    fn tokenizer(&self) -> PyTokenizer{
        PyTokenizer { inner: self.inner }
    }
}

pub PyTokenizer {
    inner: StatelessTokenizer<Arc<JapaneseDictionary>>
}

[eiennohito]

I'd like that we actually write down the list of requirements for the python binding.