RuntimeError: inconsistent tensor size

[maydaygmail]

I have problem:
load word frequency mapping - complete
loaded tensor torch.Size([798949912])
loaded tensor torch.Size([798949912, 3])
#sentences 798949912
load train data - complete
#sentences 6073
load test data - complete
Traceback (most recent call last):
File "main.py", line 195, in
train()
File "main.py", line 157, in train
for batch, item in enumerate(train_loader):
File "/home/xxxx/PyTorch_LM/lm/fast_gbw.py", line 89, in batch_generator
tracker_list[idx] = self.add(seq_length, source, target, idx, tracker)
File "/home/xxxx/lm/PyTorch_LM/lm/fast_gbw.py", line 124, in add
source[curr:batch_end, batch_idx] = self.corpus[seq_start:seq_end]
RuntimeError: inconsistent tensor size, expected tensor [19] and src [798949911] to have the same number of elements, but got 19 and 798949911 elements respectively at /pytorch/torch/lib/TH/generic/THTensorCopy.c:86

[rdspring1]

A couple of follow-up questions:

1. Did you change any of the hyper-parameters?
2. Did you use the latest version of the repo?
3. Did you build the sentence id tensor?

[zhaowei8188127]

I encountered the same problem as maydaygmail.

1. I didn't change any of the hyper-parameters.
2. Yes, I use the latest code.
3. Yes, I build the sentence id tensor using process_gbw.py.

I debuged the code, and the length returned from this line "start_idx, end_idx, length = self.sentence_id[seq_id]" is 0.

[rdspring1]

I pushed a fix for the issue. The size of the SID tensor should be the number of sentences in the corpus. It was mistakenly set to the number of words in the corpus.

[zhaowei8188127]

This issue is fixed after I pulled the newest code.
However, the loss and PPL drop so quickly when I use tied mode. Is it normal? I think the PPL should above 30.

python main.py --tied
load word frequency mapping - complete
loaded tensor torch.Size([798949912])
loaded tensor torch.Size([30301027, 2])
#sentences 30301027
load train data - complete
#sentences 6073
load test data - complete
| epoch 1 | 0/312089 batches | lr 0.10 | ms/batch 0.47 | loss 13.33 | ppl 618095.27
| epoch 1 | 1000/312089 batches | lr 0.10 | ms/batch 153.60 | loss 6.06 | ppl 427.51
| epoch 1 | 2000/312089 batches | lr 0.10 | ms/batch 153.57 | loss 3.31 | ppl 27.52
| epoch 1 | 3000/312089 batches | lr 0.10 | ms/batch 153.56 | loss 2.40 | ppl 11.04
| epoch 1 | 4000/312089 batches | lr 0.10 | ms/batch 153.64 | loss 1.94 | ppl 6.99
| epoch 1 | 5000/312089 batches | lr 0.10 | ms/batch 153.64 | loss 1.67 | ppl 5.31
| epoch 1 | 6000/312089 batches | lr 0.10 | ms/batch 153.60 | loss 1.49 | ppl 4.42
| epoch 1 | 7000/312089 batches | lr 0.10 | ms/batch 153.64 | loss 1.35 | ppl 3.87
| epoch 1 | 8000/312089 batches | lr 0.10 | ms/batch 153.64 | loss 1.26 | ppl 3.51
| epoch 1 | 9000/312089 batches | lr 0.10 | ms/batch 153.63 | loss 1.18 | ppl 3.25
| epoch 1 | 10000/312089 batches | lr 0.10 | ms/batch 153.61 | loss 1.12 | ppl 3.06

[rdspring1]

The settings for the projection matrix didn't work well.
I changed the settings for the projection matrix from LSTM 2048->256 to LSTM 1024->256.
Also, there was another small mistake in the process_gbw script.

I ran a quick test to check everything, and the model has 63.44 perplexity after the first epoch.

[zhaowei8188127]

The size of the projection matrix seems not help the PPL drop so quickly. Can you give more details on the mistake of process_gbw script?

[rdspring1]

I forgot to fill the sid tensor, so the start_idx and length values are random.
I pushed the fix, but you'll have to rerun the process_gbw script.