IndexError in static_vectors.py

[drfraser]

Hi, is spaCy 2.0 dependent on a specific version of thinc? At some point my code was working fine and also running in multiple CPU processes. I upgraded to spaCy 2.0 (don't remember what version of thinc - whatever was required for that, so probably it got updated) and I generated some features with spaCy etc back in January I think.

Now, I had to re-run this feature creation code because of another issue and now nothing seems to work. Using version 6.10 to 6.10.2 of thinc, the code works (no crashes) when I do not parallelize the code (using Process pool etc), but switching to a pool (nothing fancy), the code crashes w/o any traceback (will update this issue when I get data). As for 6.11, it always crashes as below whether running in one process or multiple ones.

OS: OS X 10.11
Python: 3.6.4
spaCy: 2.0.9
thinc: 6.11.0

I'll try rolling back thinc farther, but will wait till you tell if I ought to change the version of spaCy used. Let me know if I should do any specific tests or debugging you need.

At this point in the process, senttext was just one sentence - I disabled using pipe() because of spaCy 2.0 crashing due to pickling issue, (I think) that I filed and you fixed - haven't tested that yet.

File "parallel_compute_pcfg.py", line 93, in process_sent
    sent_tokens = tools['nlp'](senttext)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/language.py", line 341, in __call__
    doc = proc(doc)
  File "pipeline.pyx", line 393, in spacy.pipeline.Tagger.__call__
  File "pipeline.pyx", line 405, in spacy.pipeline.Tagger.predict
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 161, in __call__
    return self.predict(x)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 55, in predict
    X = layer(X)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 161, in __call__
    return self.predict(x)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 293, in predict
    X = layer(layer.ops.flatten(seqs_in, pad=pad))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 161, in __call__
    return self.predict(x)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 55, in predict
    X = layer(X)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 161, in __call__
    return self.predict(x)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 125, in predict
    y, _ = self.begin_update(X, drop=None)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 377, in uniqued_fwd
    Y_uniq, bp_Y_uniq = layer.begin_update(X_uniq, drop=drop)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 61, in begin_update
    X, inc_layer_grad = layer.begin_update(X, drop=drop)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in begin_update
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in <listcomp>
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 258, in wrap
    output = func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in begin_update
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in <listcomp>
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 258, in wrap
    output = func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in begin_update
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in <listcomp>
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 258, in wrap
    output = func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in begin_update
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 176, in <listcomp>
    values = [fwd(X, *a, **k) for fwd in forward]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/api.py", line 258, in wrap
    output = func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/_classes/static_vectors.py", line 58, in begin_update
    vectors = vector_table[ids * (ids < vector_table.shape[0])]
IndexError: arrays used as indices must be of integer (or boolean) type

[trungtv]

#59

[honnibal]

I don't understand why spaCy installed this version of Thinc --- it's not marked as compatible. I've removed Thinc 6.11.0 from PyPi.

[drfraser]

I think that was my fault. I upgraded thinc to test out the fix re: FeatureExtractor / pickling and already had spaCy 2.0.9. I did not see any warnings.