AnalyzedSentence Design

[eiennohito]

Prereq of #52

We want a ponter-generic version of return container for the result of analysis.

Rust API wants references/mut references; Python binding wants Arc.

[mh-northlander]

Considering to add MorphemeList following java/python:

struct MorphemeList<'a, T> {
    dictionary: T,
    input_text: &'a str,
    path: Vec<lattice::Node>
}

And let Morpheme have reference to MorhemeList:

struct Morpheme<'a, T> {
    list: &MorphemeList<'a, T>,
    index: usize,
    word_info: Option<WordInfo>, // for lazy load
}

MorhemeList<'a, T> will be returned from tokenizer<T>.tokenize(input: &'a str).
We may change the struct for the path field later.

[eiennohito]

In this formulation will it be usable in Pyo3? Even with wrappers.

[mh-northlander]

For python side I considered to create a variant with no generic parameter:

struct PyMorphemeList {
    dictionary: Arc<JapaneseDictionary>,
    input_text: String,
    path: Vec<lattice::Node>
}

struct PyMorpheme {
    list: Arc<PyMorphemeList>,
    index: usize,
    word_info: Option<PyWordInfo>
}

[eiennohito]

It would be nice if the Rust version of MorphemeList would be somehow reusable allocation-wise. At least in theory and not in the initial implementation.

For the Python that may be impossible (and Python FFI requires making copies for basically everything).