Load pre-trained weights into LookupTable?

[coodoo]

Just wondering is it possible or practical to load pre-trained weights into nn.LookupTable and use it for text classification tasks like sentiment analysis? Thanks.

[MTSranger]

Supposed you have tbl = nn.LookupTable(vocabSize, vectorDims) then tbl.weight would be the matrix of weights, and tbl.weight[i][j] would be the j-th dimension i-th weight vector. You can modify them directly.

[coodoo]

Thanks, ended up solved it with #747

[octavian-ganea]

What if the weight tensor is too big and I get cuda out of memory :

THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-2656/cutorch/lib/THC/generic/THCStorage.cu line=41 error=2 : out of memory

The issue is that this tensor has to be allocated twice , once at reading time and once at nn.LookupTable initiatlization.

[fmassa]

@octavian-ganea just convert the weights to float while at reading time, and then at nn.LookupTable you can copy the float weights directly to the cuda module

[octavian-ganea]

@fmassa : In my case I have a 4 million x 100 lookup table, so would like to load pre-initialized weights from a file directly. Currently, I am doing the following:

m = nn.LookupTable(4M,100)
m.weight = torch.load(t7_filename)

This is quite slow and occupies twice the memory of a 4M x 100 tensor, since the first line is allocating a huge tensor in memory, and the second one loads another big tensor in memory and afterwards it copies it to m.weight. I would like to have one single huge tensor allocated at a time in memory. Thanks.

[hashbangCoder]

Hi,

As a related question, this doesn't seem to work :

b = torch.ones(9,5)
l = nn.LookupTable(10,5)
l.weight:narrow(1,2,9):set(b)  --unchanged weights 

Which, I can kinda understand (the reference changes twice). My question is there any easy way of changing a subset of weights without memcopy?