spaCy tokenizer does not split correctly tokens separated by a slash (/) ending in a digit

[arimbr]

spaCy tokenizer seems not to tokenize correctly tokens separated by slash (/) when some of them end with a digit.

How to reproduce the behaviour

In [57]: import spacy
In [58]: nlp = spacy.load('fr')

In [59]: [t for t in nlp('Learn html5/css3/javascript/jquery')]
Out[59]: [Learn, html5/css3/javascript, /, jquery] # UNEXPECTED

In [60]: [t for t in nlp('Learn html/css/javascript/jquery')]
Out[60]: [Learn, html, /, css, /, javascript, /, jquery] # EXPECTED

Your Environment

• spaCy version: 2.0.11
• Platform: Linux-4.15.0-36-generic-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.5
• Models: fr, en

Related issue #891

[ines]

Merging this with the master issue in #1642!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.