CUDA-related error when trying to run with CPU

[windweller]

Hi,

I'm trying to run only on CPU. My PyTorch version is: torch (0.2.0.post2).
I used this line to initiate the model:
infersent = torch.load('infersent.allnli.pickle', map_location=lambda storage, loc: storage)

I got this warning:

SentEval/eval_models/models.py:54: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at
every call, possibly greately increasing memory usage. To compact weights again call flatten_parameters().
  sent_output = self.enc_lstm(sent_packed)[0]  # seqlen x batch x 2*nhid

And an assertion error when I call infersent.encode():

Traceback (most recent call last):
  File "infersent_run.py", line 155, in main
    results_transfer = se.eval(transfer_tasks)
  File "SentEval/senteval.py", line 56, in eval
    self.results = {x:self.eval(x) for x in name}
  File "SentEval/senteval.py", line 56, in <dictcomp>
    self.results = {x:self.eval(x) for x in name}
  File "SentEval/senteval.py", line 91, in eval
    self.results = self.evaluation.run(self.params, self.batcher)
  File "SentEval/binary.py", line 44, in run
    embeddings = batcher(params, batch)
  File "infersent_run.py", line 89, in batcher
    infersent_embed = params.infersent.encode(sentences, bsize=params.batch_size, tokenize=False)
  File "SentEval/eval_models/models.py", line 202, in encode
    batch = self.forward((batch, lengths[stidx:stidx + bsize]))
  File "SentEval/eval_models/models.py", line 54, in forward
    sent_output = self.enc_lstm(sent_packed)[0]  # seqlen x batch x 2*nhid
  File "/home/python2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/python2.7/site-packages/torch/nn/modules/rnn.py", line 162, in forward
    output, hidden = func(input, self.all_weights, hx)
  File "/home/python2.7/site-packages/torch/nn/_functions/rnn.py", line 351, in forward
    return func(input, *fargs, **fkwargs)
  File "/home/python2.7/site-packages/torch/autograd/function.py", line 284, in _do_forward
    flat_output = super(NestedIOFunction, self)._do_forward(*flat_input)
  File "/home/python2.7/site-packages/torch/autograd/function.py", line 306, in forward
    result = self.forward_extended(*nested_tensors)
  File "/home/python2.7/site-packages/torch/nn/_functions/rnn.py", line 293, in forward_extended
    cudnn.rnn.forward(self, input, hx, weight, output, hy)
  File "/home/python2.7/site-packages/torch/backends/cudnn/rnn.py", line 259, in forward
    _copyParams(weight, params)
  File "/home/python2.7/site-packages/torch/backends/cudnn/rnn.py", line 186, in _copyParams
    assert param_from.type() == param_to.type()
AssertionError

Any idea on why this is happening, and why is it still calling cudnn even though I want to run on CPU?

[windweller]

Actually I was able to fix this by adding:

infersent.use_cuda = False

It is possible that I might be using an older version of InferSent and the newer version made this line unnecessary.

[aconneau]

Indeed, the need for this line "infersent.use_cuda = False" was removed in a recent commit.
Now you just need to use ".cpu()" or ".cuda()" to switch between CPU/GPU.

If you're on CPU, you may want to try and play with the parameter k in:
torch.set_num_threads(k)
In my case, using less CPU cores than my server had made the generation of embeddings faster (from 40 to 70 sentences/s).