Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

Commit

Permalink
Update documents to sentence_splitter.py (#3023)
Browse files Browse the repository at this point in the history
* Update sentence_splitter.py

Update some legacy documents that look like copied from `WordSplitter`.

* Update sentence_splitter.py
  • Loading branch information
guoquan authored and schmmd committed Jul 9, 2019
1 parent 427996d commit 354a19b
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions allennlp/data/tokenizers/sentence_splitter.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,12 @@ class SentenceSplitter(Registrable):

def split_sentences(self, text: str) -> List[str]:
"""
Splits ``texts`` into a list of :class:`Token` objects.
Splits a ``text`` :class:`str` paragraph into a list of :class:`str`, where each is a sentence.
"""
raise NotImplementedError

def batch_split_sentences(self, texts: List[str]) -> List[List[str]]:
"""
This method lets you take advantage of spacy's batch processing.
Default implementation is to just iterate over the texts and call ``split_sentences``.
"""
return [self.split_sentences(text) for text in texts]
Expand Down Expand Up @@ -59,4 +58,7 @@ def split_sentences(self, text: str) -> List[str]:

@overrides
def batch_split_sentences(self, texts: List[str]) -> List[List[str]]:
"""
This method lets you take advantage of spacy's batch processing.
"""
return [[sentence.string.strip() for sentence in doc.sents] for doc in self.spacy.pipe(texts)]

0 comments on commit 354a19b

Please sign in to comment.