Unexpected (almost random) parses and no sentence segmentation when using GPU on Windows

[BramVanroy]

Following a discussion on Twitter about spaCy vs stanfordnlp vs spacy-stanfordnlp, I figured I'd do a speed comparison. The task is putting 1K lines through the whole pipeline. The test includes nlp() as well as nlp.pipe().

The testing scripts and data are available in this test repo: https://github.com/BramVanroy/parsers-test

I was extremely surprised by the big differences in speed (spaCy being much faster than stanfordnlp) so I went digging. I found that for the whole input in nlp() only one doc.sent sentence was created. In other words, no sentence segmentation was done. In addition, I saw extremely bad parsing results from spaCy. An example from using readlines() + nlp.pipe():

['The', 'decrease', 'in', '2008', 'primarily', 'relates', 'to', 'the', 'decrease', 'in', 'cash', 'and', 'cash', 'equivalents', '1', '.', '\n']
['VERB', 'PRON', 'PROPN', 'NOUN', 'VERB', 'ADV', 'VERB', 'NUM', 'PRON', 'NOUN', 'VERB', 'PROPN', 'PROPN', 'VERB', 'VERB', 'NOUN', 'SPACE']
['dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'ROOT', '']

['The', 'Company', "'s", 'current', 'liabilities', 'of', '&', 'euro;32.6', 'million', 'primarily', 'relate', 'to', 'deferred', 'income', 'from', 'collaborative', 'arrangements', 'and', 'trade', 'payables', '.', '\n']
['NOUN', 'VERB', 'AUX', 'NOUN', 'NOUN', 'PROPN', 'PROPN', 'PROPN', 'VERB', 'VERB', 'ADV', 'VERB', 'VERB', 'NOUN', 'NOUN', 'PROPN', 'NOUN', 'PROPN', 'VERB', 'NUM', 'NOUN', 'SPACE']
['dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'punct', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'ROOT', '']

['The', 'increase', 'in', 'deferred', 'income', 'is', 'related', 'to', 'new', 'deals', 'with', 'partners', '.', '\n']
['NOUN', 'PROPN', 'PROPN', 'VERB', 'NOUN', 'NOUN', 'NOUN', 'VERB', 'ADV', 'VERB', 'NOUN', 'VERB', 'NOUN', 'SPACE']
['dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'dep', 'punct', 'dep', 'dep', 'ROOT', '']

This is not how I've seen spaCy behave in the past (non-GPU version). The parses seem almost random, as if the model is not initialized correctly. Also note how basically all dependencies are dep. Again, the posted repo above may prove some insight. After more digging:

• I found that using the CPU version of spaCy does work as expected.
• I did a complete reinstall of CUDA and tried the supported version (9.2 per spaCy documentation). So I installed CUDA 9.2 (+patch) followed by installing the specifc cupy package cupy-cuda92 and spacy[cuda92]. Unfortunately, this leads to the same result.
• I found that it does work perfectly fine on Linux (Ubuntu 18.04) with 9.2. So this seems like a Windows issue.
• I tried installing spacy[cuda92]==2.1.8 to check whether the issue also arises in spaCy 2.1. Even though installation works, I get an error propagating from thinc (dependency issue?). Possibly related: module 'thinc_gpu_ops' has no attribute 'hash' thinc#79

  File "C:/dev/python/parsers-test/spacy_test.py", line 7, in parse
    doc = nlp(text)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\spacy\language.py", line 402, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))
  File "pipes.pyx", line 392, in spacy.pipeline.pipes.Tagger.__call__
  File "pipes.pyx", line 411, in spacy.pipeline.pipes.Tagger.predict
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\model.py", line 169, in __call__
    return self.predict(x)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\feed_forward.py", line 40, in predict
    X = layer(X)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\model.py", line 169, in __call__
    return self.predict(x)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 310, in predict
    X = layer(layer.ops.flatten(seqs_in, pad=pad))
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\model.py", line 169, in __call__
    return self.predict(x)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\feed_forward.py", line 40, in predict
    X = layer(X)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\model.py", line 169, in __call__
    return self.predict(x)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\model.py", line 133, in predict
    y, _ = self.begin_update(X, drop=None)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 379, in uniqued_fwd
    Y_uniq, bp_Y_uniq = layer.begin_update(X_uniq, drop=drop)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\feed_forward.py", line 46, in begin_update
    X, inc_layer_grad = layer.begin_update(X, drop=drop)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 163, in begin_update
    values = [fwd(X, *a, **k) for fwd in forward]
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 163, in <listcomp>
    values = [fwd(X, *a, **k) for fwd in forward]
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 256, in wrap
    output = func(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 163, in begin_update
    values = [fwd(X, *a, **k) for fwd in forward]
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 163, in <listcomp>
    values = [fwd(X, *a, **k) for fwd in forward]
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 256, in wrap
    output = func(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 163, in begin_update
    values = [fwd(X, *a, **k) for fwd in forward]
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 163, in <listcomp>
    values = [fwd(X, *a, **k) for fwd in forward]
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\api.py", line 256, in wrap
    output = func(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\parsers-test-NtX--KFc\lib\site-packages\thinc\neural\_classes\hash_embed.py", line 59, in begin_update
    keys = self.ops.hash(ids, self.seed) % self.nV
  File "ops.pyx", line 967, in thinc.neural.ops.CupyOps.hash
AttributeError: module 'thinc_gpu_ops' has no attribute 'hash'

Info about spaCy

• spaCy version: 2.2.3
• Platform: Windows-10-10.0.18362-SP0
• Python version: 3.7.5
• Models: en, nl

[honnibal]

Hey,

So I'm not 100% sure what the problem is here, or actually which part you're identifying as an issue. You've raised 2x2x2 factors in an unhelpful way: parse quality vs speed, spaCy and StanfordNLP, .pipe() vs .__call__. If you think the issue is in the StanfordNLP wrapper you should raise an issue at the https://github.com/explosion/spacy-stanfordnlp repo. If it isn't related to the StanfordNLP wrapper, that part isn't relevant. If it's just a matter of the model, it's not an issue of either the wrapper or spaCy. We're also not responsible for making the StanfordNLP models faster.

Running those sentences with the en_core_web_sm, the accuracy seems fine and we can't reproduce the parses you've pasted.

Finally, it's hard not to bristle at the title of your issue. Like, cmon!

[BramVanroy]

@honnibal Ho there, no need to take offence. I use abysmal here to highlight how unexpected this is, and to show that something seems off. Perhaps that was not the right word to choose. My apologies. Have you taken a look at the examples that I give? They are abysmal, there is no other word for it. They are not "just a selection" either, the whole output looks like this. I'm sure that if you look at it objectively you cannot but agree that they are. And I am saying that because I have never seen spaCy behave in this way. That being said, this is the first time I'm using spacy[cuda] so that might relate to the problem.

I am raising this issue not because I think spaCy is bad, of course. I have shown time and time again that I love it and use it almost daily. I raise this issue because in this case something seems wrong. Just look at the examples. Those are not reasonable at all and I think something must be wrong but I can't pinpoint what.

Again, please don't feel offended as that was not at all the goal of my post. I appreciate all the work you guys have done (as I've said already numerous times). I call it abysmal because this is not the output that spaCy would normally give and something seems wrong. Especially if you also take the failure of segmentation in consideration.

I hope we're good.

Please re-open. I'll improve my OP.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.