RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

[MuruganR96]

sir i was pretrained for our BERT-Base model for Multi-GPU training 8 GPUs. preprocessing succeed but next step training it shown error. in run_lm_finetuning.py.

python3 run_lm_finetuning.py --bert_model bert-base-uncased --do_train --train_file vocab007.txt --output_dir models --num_train_epochs 5.0 --learning_rate 3e-5 --train_batch_size 32 --max_seq_length 128

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 646, in <module>
    main()
  File "run_lm_finetuning.py", line 594, in main
    loss = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/pytorch_pretrained_bert/modeling.py", line 695, in forward
    output_all_encoded_layers=False)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/pytorch_pretrained_bert/modeling.py", line 626, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/newvolume/pytorch_bert_env/lib/python3.5/site-packages/pytorch_pretrained_bert/modeling.py", line 187, in forward
    seq_length = input_ids.size(1)
RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Thanks.

[MuruganR96]

Sir how to resolve this? i am beginner for pytorch.
Thanks.

[thomwolf]

I will have a look, I am not familiar with run_lm_finetuning yet.
In the meantime maybe @tholor has an advice?

[tholor]

Haven't seen this error before, but how does your training corpus "vocab007.txt" look like? Is training working successfully for the file "samples/sample_text.txt"?

[MuruganR96]

Sir my vocab007.txt is a my own text sentences same as samples/sample_text.txt. but I won't tested before this sample _text.txt. directly i am put training to my vocab007.txt
Thank you so much @thomwolf @tholor sir.

[tholor]

Not sure if I understood your last message. Is this solved?

[MuruganR96]

@tholor sir, stil now i am not solving this issue.

Is training working successfully for the file "samples/sample_text.txt"?

No, i am not train the file "samples/sample_text.txt"

how does your training corpus "vocab007.txt" look like?

this is line by line sentence like file "samples/sample_text.txt"

sir i think this shape issue. batch vice split datas for multi gpu. that time this issue occurred.

sir any suggestion? how to resolve is bug.

thanks.

[tholor]

I cannot reproduce your error. Just tested again with a dummy corpus (referenced in the readme) on a 4x P100 machine using the same parameters as you:

python3 run_lm_finetuning.py --bert_model bert-base-uncased --do_train --train_file ../samples/small_wiki_sentence_corpus.txt --output_dir models --num_train_epochs 1.0 --learning_rate 3e-5 --train_batch_size 32 --max_seq_length 128

Training just started normally. I would recommend to:

1. Check your local setup and try to run with the above corpus (download here). If this doesn't work, there's something wrong with your setup (e.g. CUDA)
2. If 1 works, examine your training corpus "vocab007.txt". I suppose there's something wrong here causing wrong input_ids. A good starting point will be the logs that you see in the beginning of model training and print some training examples (including input_ids). They should look something like this:

01/11/2019 08:20:24 - INFO - __main__ -   ***** Running training *****
01/11/2019 08:20:24 - INFO - __main__ -     Num examples = 476462
01/11/2019 08:20:24 - INFO - __main__ -     Batch size = 32
01/11/2019 08:20:24 - INFO - __main__ -     Num steps = 14889
Epoch:   0%|                                                                                                                                                                                          | 0/1 [00:00<?, ?it/s]
01/11/2019 08:20:24 - INFO - __main__ -   *** Example ***                                                                                                                                         | 0/14890 [00:00<?, ?it/s]
01/11/2019 08:20:24 - INFO - __main__ -   guid: 0
01/11/2019 08:20:24 - INFO - __main__ -   tokens: [CLS] [MASK] jp ##g mini ##at ##ur [MASK] [UNK] [MASK] eine ##m [MASK] [UNK] mit [UNK] . [SEP] [UNK] [UNK] [MASK] jp ##g mini ##at ##ur [UNK] [MASK] [MASK] ##t eine [UNK]
 in [UNK] [UNK] - [UNK] [UNK] [UNK] [UNK] in [UNK] [SEP]
01/11/2019 08:20:24 - INFO - __main__ -   input_ids: 101 103 16545 2290 7163 4017 3126 103 100 103 27665 2213 103 100 10210 100 1012 102 100 100 103 16545 2290 7163 4017 3126 100 103 103 2102 27665 100 1999 100 100 1011 
100 100 100 100 1999 100 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/11/2019 08:20:24 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/11/2019 08:20:24 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/11/2019 08:20:24 - INFO - __main__ -   LM label: [-1, 1012, -1, -1, -1, -1, -1, 100, -1, 1999, -1, -1, 100, -1, -1, -1, -1, -1, -1, -1, 1012, -1, -1, -1, -1, -1, -1, 3413, 3771, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
 -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
 -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
01/11/2019 08:20:24 - INFO - __main__ -   Is next sentence label: 0

[MuruganR96]

Check your local setup and try to run with the above corpus

ok @tholor sir, now i will check.

[yuhui-zh15]

Hi, it can be solved by using python3.6.

See #184

[MuruganR96]

Thanks sir.

[thomwolf]

Fixed on master now (compatible with Python 3.5 again)