Option to "disable sentence segmentation" needed

[KoichiYasuoka]

I used "lzh" model, but its performance for sentence segmentation seems rather worse. So I tried to disable sentence segmentation:

import spacy_udpipe
class M(spacy_udpipe.UDPipeModel):
  def tokenize(self,text:str):
    t=self.model.newTokenizer(self.model.TOKENIZER_PRESEGMENTED)
    return self._read(text=text,input_format=t)
m=M("lzh")
lzh=spacy_udpipe.UDPipeLanguage(m,m._meta)
doc=lzh("不入虎穴不得虎子")

This quick hack works well, and I think we need an option for spacy_udpipe.load to disable sentence segmentation. How do you think, @asajatovic ?

[asajatovic]

@KoichiYasuoka after some consideration, I think it could work as there are a few ancient languages that could benefit from pre-segmented text input.
What concerns me is how the end-user will be able to provide pre-segmented text required for this. I'd like to know what you think about this?
Unfortunately, I am too busy to work on this, so I'd like to encourage you to do it (if you are up for it)! 😄

[asajatovic]

@KoichiYasuoka, I initially thought it would be much harder to enable than it was in #19.
It works now! 😅