SudachiPy compatibility

[mh-northlander]

Part of #25.
Provide same interface to SudachiPy as much as possible.

Maybe after rust API become stable (#28)

[mh-northlander]

SudachiPy API to implement

• Dictionary
  • __new__(config_path=None, resource_dir=None, dict_type=None)
  • create(self, mode=None) -> Tokenizer
  • close(self)
• Tokenizer
  • tokenize(self, text: str, mode=None, logger=None) -> MorphemeList
• MorphemeList
  • behave as List[Morpheme]
    • __getitem__, __len__, __iter__
  • __str__
    • "".join([morpheme.surface() for morpheme in self])
  • empty(cls) -> MorphemeList
    • generates an empty MorphemeList
  • get_internal_cost(self)
    • returns the total cost of this path
  • size(self)
    • alias of __len__
• Morpheme
  • getter functions
    • begin, end
      • returns the index in the input text
    • surface, part_of_speech, part_of_speech_id
    • dictionary_form, normalized_form, reading_form
    • is_oov, word_id, dictionary_id, synonym_group_ids
  • split(self, mode) -> MophemeList
    • splits this morpheme based on mode
  • get_word_info(self) -> WordInfo
• WordInfo
  • getter
    • surface, head_word_length, pos_id
    • normalized_form, dictionary_form_id, dictionary_form_id, reading_form
    • a_unit_split, b_unit_split, word_structure, synonym_group_ids
  • length(self)
    • shorthand for head_word_length

[eiennohito]

• Dictonary - OK
• Tokeniser - OK
• Morpheme / WordInfo - from the binding point of view they probably should be aliases if it is possible, maybe with lazy loading of dictionary information eventually
  • Otherwise there would be a lot of Arc internally in the morheme (or it is OK?)

[mh-northlander]

Differences left:
Dictionary.close() is not implemented.
Dictionary.__new__(-) should take dict_type (#73)
Tokenizer.tokenize(-) takes enable_debug flag instead of logger.
MorphemeList.empty(dict: Dictionary) takes dictionary as args.
SplitMode locates at the top level (not under Tokenizer).

[eiennohito]

The most problematic thing will be passing logger to the tokenizer. I'm not sure if any actual user uses it (the options seems mostly for debugging purposes).

[eiennohito]

May have comments from GinZa, but will create new issues for them

[hiroshi-matsuda-rit]

I did not find any problematic points for GiNZA in this issue.
Please go ahead!