Tokens can be lemmatized into the empty string

[mlehl88]

It seems like the token 's' can sometimes become lemmatized into the empty string.

>>> from spacy.en import English
>>> nlp = English()
>>> my_string = """s..."""
>>> tokens = nlp(my_string)
>>> for t in tokens:
...     print("token: %s, lemma: %s" % (t, t.lemma_))
... 
token: s, lemma: 
token: ..., lemma: ...

In contrast, this does not happen if the string 's' is not followed by ellipsis dots:

>>> my_string = """s"""
>>> tokens = nlp(my_string)
>>> for t in tokens:
...     print("token: %s, lemma: %s" % (t, t.lemma_))
... 
token: s, lemma: s

This behaviour might cause unexpected results in downstream applications. One consequence is that textacy sometimes extracts the empty string as a keyword.

Environment

• Operating System: Mac OS X Version 10.10.5
• Python Version Used: 3.6.0
• spaCy Version Used: 1.5
• Environment Information: Anaconda virtual environment
• Spacy model: en-1.1.0

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.