Minor tokenization issue with lowercase 'i' + contraction

[NSchrading]

spaCy correctly tokenizes capital "I" + contraction ('d, 'm, 'll, 've) e.g.:

from spacy.en import English
nlp = English()
tok = nlp("I'm")
print([x.lower_ for x in tok])

>>> ['i', "'m"]

but when the "I" is a lowercase ("i") it does not tokenize into two tokens:

from spacy.en import English
nlp = English()
tok = nlp("i'm")
print([x.lower_ for x in tok])

>>> ["i'm"]

Not a big deal, and this may be the intent, since we don't know if the user meant capital "I", but I can't think of any problems that would happen if it tokenized the lowercase version into two.

[honnibal]

Thanks, this is a gap in the tokenization special-case data. I'll fix this. It should also handle stuff like "im".

[honnibal]

Fixed in version 0.70.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.