[Question] Sentence segmentation in gensim

[narenchoudhary]

I need to split corpus text stream into sentences for further processing. I checked gensim documentation but could not find anything on sentence segmentation.

Is there any utility available in gensim for sentence segmentation?

[gojomo]

No, gensim generally expects such tokenization to happen elsewhere. Popular Python options for this include NLTK or Spacy.io.

[narenchoudhary]

It's a very basic and must have functionality in a nlp library. gensim provides functions for parsing text from corpus. It also has models that take list of sentences as argument. But there is no utility for segmenting text into sentences which is very disappointing. 😞

[piskvorky]

I agree. Unsupervised segmentation (into blocks/sentences/words) falls nicely under gensim's mission. Pull requests welcome :)

CC @tmylk .

[tmylk]

Added to wiki

[rpedela]

The summarization code splits text into sentences. Based on the English summary results, it does a great job.

https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/summarization/textcleaner.py