Tokenization not working using v2.1

[rulai-huajunzeng]

How to reproduce the behaviour

I found a bug where tokenization is completely not working with version 2.1.0a10 on python 2.7. I have reproduced this on three of my machines.

$ conda create -n py27_spacy2 python=2.7
$ source activate py27_spacy2
$ pip install -U spacy-nightly
$ python -m spacy download en_core_web_sm
$ python -c "import spacy; nlp=spacy.load('en_core_web_sm'); doc=nlp(u'hello world'); print ','.join([t.text for t in doc])"
h,e,ll,o,w,o,r,l,d

Your Environment

• Operating System: Ubuntu
• Python Version Used: 2.7
• spaCy Version Used: 2.1.0a10

[honnibal]

Hmm that's very confusing. I don't see how our tests could be passing if this is the case. I don't suppose you're somehow getting a narrow unicode build of Python?

Another thing to check is that your $ pip install -U spacy-nightly line is actually running the version of pip you expect. But then if it werent't I don't see how spaCy would be in your environment. Hmm.

[rulai-huajunzeng]

It's quite easy to reproduce for me. The following is another run. I also checked my python and pip and they are correct.

user@alienware:~/dev$ conda --version
conda 4.6.1

user@alienware:~/dev$ conda create -y -n py27_spacy2 python=2.7
Collecting package metadata: done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.6.1
  latest version: 4.6.7

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /home/user/miniconda3/envs/py27_spacy2

  added / updated specs:
    - python=2.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2019.1.23  |                0         126 KB
    certifi-2018.11.29         |           py27_0         146 KB
    openssl-1.1.1b             |       h7b6447c_0         4.0 MB
    pip-19.0.3                 |           py27_0         1.8 MB
    python-2.7.15              |       h9bab390_6        12.8 MB
    setuptools-40.8.0          |           py27_0         635 KB
    wheel-0.33.1               |           py27_0          39 KB
    ------------------------------------------------------------
                                           Total:        19.5 MB

The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/linux-64::ca-certificates-2019.1.23-0
  certifi            pkgs/main/linux-64::certifi-2018.11.29-py27_0
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1b-h7b6447c_0
  pip                pkgs/main/linux-64::pip-19.0.3-py27_0
  python             pkgs/main/linux-64::python-2.7.15-h9bab390_6
  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-64::setuptools-40.8.0-py27_0
  sqlite             pkgs/main/linux-64::sqlite-3.26.0-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.33.1-py27_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3



Downloading and Extracting Packages
python-2.7.15        | 12.8 MB   | #################################################################################################################### | 100% 
certifi-2018.11.29   | 146 KB    | #################################################################################################################### | 100% 
pip-19.0.3           | 1.8 MB    | #################################################################################################################### | 100% 
openssl-1.1.1b       | 4.0 MB    | #################################################################################################################### | 100% 
ca-certificates-2019 | 126 KB    | #################################################################################################################### | 100% 
wheel-0.33.1         | 39 KB     | #################################################################################################################### | 100% 
setuptools-40.8.0    | 635 KB    | #################################################################################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate py27_spacy2
#
# To deactivate an active environment, use:
# > source deactivate
#

user@alienware:~/dev$ source activate py27_spacy2
(py27_spacy2) user@alienware:~/dev$ pip install -U spacy-nightly
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting spacy-nightly
  Downloading https://files.pythonhosted.org/packages/e3/18/000e94b05aa86b8a414ad34c043fd30d242a54b170363f13f53a92036f2b/spacy_nightly-2.1.0a10-cp27-cp27mu-manylinux1_x86_64.whl (27.6MB)
    100% |████████████████████████████████| 27.6MB 1.4MB/s 
Collecting thinc<7.1.0,>=7.0.2 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/7f/f5/e37a2e5167be02b417046fd00980d1e74bce02d0dff635416cb2c75e8286/thinc-7.0.2-cp27-cp27mu-manylinux1_x86_64.whl (2.0MB)
    100% |████████████████████████████████| 2.1MB 8.3MB/s 
Collecting pathlib==1.0.1; python_version < "3.4" (from spacy-nightly)
Collecting blis<0.3.0,>=0.2.2 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/d7/46/19c4b2cef1c210e99db89bc0e2fc591c0d5e6fb230aee54b5993c57f0b73/blis-0.2.3.tar.gz (1.5MB)
    100% |████████████████████████████████| 1.5MB 7.5MB/s 
Collecting plac<1.0.0,>=0.9.6 (from spacy-nightly)
  Using cached https://files.pythonhosted.org/packages/9e/9b/62c60d2f5bc135d2aa1d8c8a86aaf84edb719a59c7f11a4316259e61a298/plac-0.9.6-py2.py3-none-any.whl
Collecting wasabi<1.1.0,>=0.0.12 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/8d/aa/41ecccdbf9a0b1a5f9a9cc54fc72e4744eb6b86ed73bc4b3ca941cad945d/wasabi-0.1.2.tar.gz
Collecting srsly<1.1.0,>=0.0.5 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/db/65/066ab96cc14ccc2fe908663fab329df96bd92a168f74d6249473b4fb0a55/srsly-0.0.5-cp27-cp27mu-manylinux1_x86_64.whl (174kB)
    100% |████████████████████████████████| 184kB 11.9MB/s 
Collecting numpy>=1.15.0 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/c4/33/8ec8dcdb4ede5d453047bbdbd01916dbaccdb63e98bba60989718f5f0876/numpy-1.16.2-cp27-cp27mu-manylinux1_x86_64.whl (17.0MB)
    100% |████████████████████████████████| 17.0MB 1.9MB/s 
Collecting murmurhash<1.1.0,>=0.28.0 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/ed/31/247b34db5ab06afaf5512481e77860fb4cd7a0c0ddff9d2566651c8c2f07/murmurhash-1.0.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting requests<3.0.0,>=2.13.0 (from spacy-nightly)
  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting jsonschema<3.0.0,>=2.6.0 (from spacy-nightly)
  Using cached https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting cymem<2.1.0,>=2.0.2 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/df/b1/4ff2cbd423184bd68e85f1daa6692753cd7710b0ba68552eb64542906a57/cymem-2.0.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting preshed<2.1.0,>=2.0.1 (from spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/25/b1/9098d07e70b960001a8a9b99435c6987006d0d7bcbf20523adce9272f66e/preshed-2.0.1-cp27-cp27mu-manylinux1_x86_64.whl (80kB)
    100% |████████████████████████████████| 81kB 34.3MB/s 
Collecting tqdm<5.0.0,>=4.10.0 (from thinc<7.1.0,>=7.0.2->spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/6c/4b/c38b5144cf167c4f52288517436ccafefe9dc01b8d1c190e18a6b154cd4a/tqdm-4.31.1-py2.py3-none-any.whl (48kB)
    100% |████████████████████████████████| 51kB 39.1MB/s 
Collecting thinc-gpu-ops<0.1.0,>=0.0.1 (from thinc<7.1.0,>=7.0.2->spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/a4/ad/11ab80a24bcedd7dd0cfabaedba2ceaeca11f1aaeeff432a3d2e63ca7d02/thinc_gpu_ops-0.0.4.tar.gz (483kB)
    100% |████████████████████████████████| 491kB 7.1MB/s 
Collecting urllib3<1.25,>=1.21.1 (from requests<3.0.0,>=2.13.0->spacy-nightly)
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0.0,>=2.13.0->spacy-nightly)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests<3.0.0,>=2.13.0->spacy-nightly)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /home/user/miniconda3/envs/py27_spacy2/lib/python2.7/site-packages (from requests<3.0.0,>=2.13.0->spacy-nightly) (2018.11.29)
Collecting functools32; python_version == "2.7" (from jsonschema<3.0.0,>=2.6.0->spacy-nightly)
  Downloading https://files.pythonhosted.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz
Building wheels for collected packages: blis, wasabi, thinc-gpu-ops, functools32
  Building wheel for blis (setup.py) ... done
  Stored in directory: /home/user/.cache/pip/wheels/d8/bc/d8/fa2e772b04b368972ed51759dc69275e020649a14ff30cf853
  Building wheel for wasabi (setup.py) ... done
  Stored in directory: /home/user/.cache/pip/wheels/64/6b/79/6d2b350fba2e7fd0ec125a929c8b67b022731bea36a87bcc87
  Building wheel for thinc-gpu-ops (setup.py) ... done
  Stored in directory: /home/user/.cache/pip/wheels/eb/ba/a3/9af9f326ed0d75a4540378af64a05a0e42be39d9b8513f3aea
  Building wheel for functools32 (setup.py) ... done
  Stored in directory: /home/user/.cache/pip/wheels/b5/18/32/77a1030457155606ba5e3ec3a8a57132b1a04b1c4f765177b2
Successfully built blis wasabi thinc-gpu-ops functools32
Installing collected packages: pathlib, tqdm, numpy, blis, thinc-gpu-ops, plac, srsly, murmurhash, cymem, preshed, wasabi, thinc, urllib3, chardet, idna, requests, functools32, jsonschema, spacy-nightly
Successfully installed blis-0.2.3 chardet-3.0.4 cymem-2.0.2 functools32-3.2.3.post2 idna-2.8 jsonschema-2.6.0 murmurhash-1.0.2 numpy-1.16.2 pathlib-1.0.1 plac-0.9.6 preshed-2.0.1 requests-2.21.0 spacy-nightly-2.1.0a10 srsly-0.0.5 thinc-7.0.2 thinc-gpu-ops-0.0.4 tqdm-4.31.1 urllib3-1.24.1 wasabi-0.1.2
(py27_spacy2) user@alienware:~/dev$ python -m spacy download en_core_web_sm
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting en_core_web_sm==2.1.0a7 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.1.0a7/en_core_web_sm-2.1.0a7.tar.gz#egg=en_core_web_sm==2.1.0a7
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.1.0a7/en_core_web_sm-2.1.0a7.tar.gz (11.0MB)
    100% |████████████████████████████████| 11.1MB 10.6MB/s 
Installing collected packages: en-core-web-sm
  Running setup.py install for en-core-web-sm ... done
Successfully installed en-core-web-sm-2.1.0a7
✔ Linking successful
/home/user/miniconda3/envs/py27_spacy2/lib/python2.7/site-packages/en_core_web_sm
-->
/home/user/miniconda3/envs/py27_spacy2/lib/python2.7/site-packages/spacy/data/en_core_web_sm
You can now load the model via spacy.load('en_core_web_sm')
(py27_spacy2) user@alienware:~/dev$ python -c "import spacy; nlp=spacy.load('en_core_web_sm'); doc=nlp(u'hello world'); print ','.join([t.text for t in doc])"
h,e,ll,o,w,o,r,l,d
(py27_spacy2) user@alienware:~/dev$ python -c "import sys; print sys.maxunicode"
1114111
(py27_spacy2) user@alienware:~/dev$ which pip
/home/user/miniconda3/envs/py27_spacy2/bin/pip
(py27_spacy2) user@alienware:~/dev$ which python
/home/user/miniconda3/envs/py27_spacy2/bin/python

[honnibal]

Okay, thanks! Definitely seems bad --- will look into it.

[svlandeg]

I can't reproduce either :(
@rulai-huajunzeng : what happens if you use the default English model instead?

from spacy.lang.en import English
nlp = English()

[hzeng-otterai]

Using the default English model the result is correct. But using spacy.load('en_core_web_sm') the result is the same. I am guessing maybe there are conflicts with previous installed spacy models?

>>> from spacy.lang.en import English
>>> nlp = English()
>>> doc=nlp(u'hello world')
>>> print ','.join([t.text for t in doc])
hello,world
>>> import spacy
>>> nlp=spacy.load('en_core_web_sm')
>>> doc=nlp(u'hello world')
>>> print ','.join([t.text for t in doc])
h,e,ll,o,w,o,r,l,d
>>> 

[svlandeg]

@rulai-huajunzeng @handsomezebra : Any chance you could run this entire sentence through the nlp pipeline?
Well I wonder how this will / shall look after tokenization with the model - ill or not ?

[rulai-huajunzeng]

>>> from spacy.lang.en import English
>>> nlp = English()
>>> doc=nlp(u'Well I wonder how this will / shall look after tokenization with the model - ill or not ?')
>>> print ','.join([t.text for t in doc])
Well,I,wonder,how,this,will,/,shall,look,after,tokenization,with,the,model,-,ill,or,not,?
>>> import spacy
>>> nlp=spacy.load('en_core_web_sm')
>>> doc=nlp(u'Well I wonder how this will / shall look after tokenization with the model - ill or not ?')
>>> print ','.join([t.text for t in doc])
W,e,l,l,I,w,o,n,d,e,r,h,o,w,t,h,i,s,w,i,l,l,/,s,h,a,l,l,l,o,o,k,a,f,t,e,r,t,o,k,e,n,i,z,a,t,i,o,n,w,i,t,h,t,h,e,m,o,d,e,l,-,i,ll,o,r,n,o,t,?

[ines]

Thanks for your patience on this, this is definitely mysterious!

And yeah, maybe what you're loading as en_core_web_sm here isn't actually the correct model package? Could you print nlp.meta and see what that returns?

[rulai-huajunzeng]

>>> import spacy
>>> nlp=spacy.load('en_core_web_sm')
>>> print nlp.meta
{u'description': u'English multi-task CNN trained on OntoNotes. Assigns context-specific token vectors, POS tags, dependency parse and named entities.', u'sources': [u'OntoNotes 5'], u'speed': {u'gpu': None, u'nwords': 291344, u'cpu': 7436.9320951768}, u'lang': u'en', u'pipeline': [u'tagger', u'parser', u'ner'], u'name': u'core_web_sm', u'license': u'MIT', u'author': u'Explosion AI', u'url': u'https://explosion.ai', u'vectors': {u'keys': 0, u'width': 0, u'vectors': 0, u'name': None}, u'version': u'2.1.0a7', u'spacy_version': u'>=2.1.0a4', u'parent_package': u'spacy-nightly', u'email': u'contact@explosion.ai', u'accuracy': {u'token_acc': 99.0646795541, u'ents_p': 85.4670314332, u'ents_r': 85.5530095568, u'uas': 91.5689404737, u'tags_acc': 96.8320691599, u'ents_f': 85.5099988828, u'las': 89.682797403}}

[rulai-huajunzeng]

I still got the same issue even using the stable release of 2.1.0