Training crashes at end

[devinbostIL]

I setup a conda environment with PyTorch 0.4 and installed this fork of OpenNMT-py, as I mentioned in the original issue I had posted here: OpenNMT#743

I then ran:

python preprocess.py -train_src C:\src\torchevere-offensive-classifier\training\character\train_src.txt -train_tgt C:\src\torchevere-offensive-classifier\training\character\train_dst.txt -valid_src C:\src\torchevere-offensive-classifier\training\character\val_src.txt -valid_tgt C:\src\torchevere-offensive-classifier\training\character\val_dst.txt -save_data data/character/tc-offense-classifier-character_v3 -src_seq_length 5000 -tgt_seq_length 5000

python train.py -data data/character/tc-offense-classifier-character_v3 -save_model tc-offense-classifier-character_v3 -gpuid 0 -layers 3 -learning_rate_decay 0.99 -train_steps 10000 -rnn_size 500

Everything ran smoothly until it crashed at the end.
Here's the output:

(pyTorchOffensive) C:\src\pyopennmt\ubiqus\OpenNMT-py>python train.py -data data/character/tc-offense-classifier-character_v3 -save_model tc-offense-classifier-character_v3 -gpuid 0 -layers 3 -learning_rate_decay 0.99 -train_steps 10000 -rnn_size 500
		Loading train dataset from data/character\tc-offense-classifier-character_v3.train.1.pt, number of examples: 3384
		 * vocabulary size. source = 165; target = 6
		Building model...
		Intializing model parameters.
		NMTModel(
		  (encoder): RNNEncoder(
		    (embeddings): Embeddings(
		      (make_embedding): Sequential(
		        (emb_luts): Elementwise(
		          (0): Embedding(165, 500, padding_idx=1)
		        )
		      )
		    )
		    (rnn): LSTM(500, 500, num_layers=3, dropout=0.3)
		  )
		  (decoder): InputFeedRNNDecoder(
		    (embeddings): Embeddings(
		      (make_embedding): Sequential(
		        (emb_luts): Elementwise(
		          (0): Embedding(6, 500, padding_idx=1)
		        )
		      )
		    )
		    (dropout): Dropout(p=0.3)
		    (rnn): StackedLSTM(
		      (dropout): Dropout(p=0.3)
		      (layers): ModuleList(
		        (0): LSTMCell(1000, 500)
		        (1): LSTMCell(500, 500)
		        (2): LSTMCell(500, 500)
		      )
		    )
		    (attn): GlobalAttention(
		      (linear_in): Linear(in_features=500, out_features=500, bias=False)
		      (linear_out): Linear(in_features=1000, out_features=500, bias=False)
		      (softmax): Softmax()
		      (tanh): Tanh()
		    )
		  )
		  (generator): Sequential(
		    (0): Linear(in_features=500, out_features=6, bias=True)
		    (1): LogSoftmax()
		  )
		)
		* number of parameters: 13862506
		encoder:  6094500
		decoder:  7768006
		Making optimizer for training.
		
		Start training...
		Loading train dataset from data/character\tc-offense-classifier-character_v3.train.1.pt, number of examples: 3384
		Step 50, 10000; acc:  27.34; ppl:  16.07; xent:   2.78; lr: 1.00000; 14099 / 3200 tok/s;      4 sec
		GPU 0: for information we completed an epoch at step 54
		Loading train dataset from data/character\tc-offense-classifier-character_v3.train.1.pt, number of examples: 3384
		Step 100, 10000; acc:  74.22; ppl:   1.45; xent:   0.37; lr: 1.00000; 26876 / 2614 tok/s;      6 sec
		GPU 0: for information we completed an epoch at step 107
		
		. . . 
		
		Loading train dataset from data/character\tc-offense-classifier-character_v3.train.1.pt, number of examples: 3384
		Step 9950, 10000; acc: 100.00; ppl:   1.00; xent:   0.00; lr: 1.00000; 16583 / 2462 tok/s;    616 sec
		GPU 0: for information we completed an epoch at step 9965
		Loading train dataset from data/character\tc-offense-classifier-character_v3.train.1.pt, number of examples: 3384
		Step 10000, 10000; acc: 100.00; ppl:   1.00; xent:   0.00; lr: 1.00000; 19345 / 2416 tok/s;    619 sec
		Loading valid dataset from data/character\tc-offense-classifier-character_v3.valid.1.pt, number of examples: 376
		Traceback (most recent call last):
		  File "train.py", line 41, in <module>
		    main(opt)
		  File "train.py", line 28, in main
		    single_main(opt)
		  File "C:\src\pyopennmt\ubiqus\OpenNMT-py\train_single.py", line 120, in main
		    opt.valid_steps)
		  File "C:\src\pyopennmt\ubiqus\OpenNMT-py\onmt\trainer.py", line 176, in train
		    valid_stats = self.validate(valid_iter)
		  File "C:\src\pyopennmt\ubiqus\OpenNMT-py\onmt\trainer.py", line 208, in validate
		    for batch in valid_iter:
		  File "C:\src\pyopennmt\ubiqus\OpenNMT-py\onmt\inputters\inputter.py", line 423, in __iter__
		    for batch in self.cur_iter:
		  File "C:\Anaconda3\envs\pyTorchOffensive\lib\site-packages\torchtext\data\iterator.py", line 151, in __iter__
		    self.train)
		  File "C:\Anaconda3\envs\pyTorchOffensive\lib\site-packages\torchtext\data\batch.py", line 27, in __init__
		    setattr(self, name, field.process(batch, device=device, train=train))
		  File "C:\Anaconda3\envs\pyTorchOffensive\lib\site-packages\torchtext\data\field.py", line 188, in process
		    tensor = self.numericalize(padded, device=device, train=train)
		  File "C:\Anaconda3\envs\pyTorchOffensive\lib\site-packages\torchtext\data\field.py", line 287, in numericalize
		    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
		  File "C:\Anaconda3\envs\pyTorchOffensive\lib\site-packages\torchtext\data\field.py", line 287, in <listcomp>
		    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
		  File "C:\Anaconda3\envs\pyTorchOffensive\lib\site-packages\torchtext\data\field.py", line 287, in <listcomp>
		    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
		KeyError: '🏻'

Luckily, there was still a model file that was produced during the middle of the training (half way through the specified number of training steps), and I was able to use that with translate.py. And thankfully, it solved the issue with having blanks in the output file, as I mentioned in that original issue (OpenNMT#743).

However, it's still an issue to have the training crash at the end of the process, so I'm reporting this bug.

[devinbostIL]

Update: I spoke a little too soon on the blanks issue. There are still a very small number of blanks in the output file. However, it's a major improvement. Instead of nearly half of the outputs being blank, out of 1374971 input lines, only 211 resulted in blank values.

[vince62s]

yes we faced a similar issue. see pytorch/text#337

I was able to train a big model without any issue but obviously in some situation this is happening.
We'll try to look into it.

[vince62s]

@devinbostIL can you pull and try again ?
thanks

[hughperkins]

Might be working now :)

$ python train.py -data data/demo -save_model demo-model -gpuid 0 -train_steps 100
Start training...
Loading train dataset from data/demo.train.1.pt, number of examples: 10000
Step 50,   100; acc:   6.27; ppl: 10845.86; xent:   9.29; lr: 1.00000; 4726 / 5988 tok/s;     13 sec
Step 100,   100; acc:   6.66; ppl: 271385.80; xent:  12.51; lr: 1.00000; 6241 / 6769 tok/s;     26 sec
GPU 0: for information we completed an epoch at step 101
Loading train dataset from data/demo.train.1.pt, number of examples: 10000

(whereas before I got File "/toknas/hugh/condap4/lib/python3.6/site-packages/torchtext/data/field.py", line 287, in <listcomp> arr = [[self.vocab.stoi[x] for x in ex] for ex in arr] KeyError: 'notebooks' , but that was training over 10,000 steps, which I havent tried yet)

[hughperkins]

Update: seems like saving is working without crashing for me now. Not sure if I was encountering the same issue as @devinbostIL ?

GPU 0: for information we completed an epoch at step 9892
Loading train dataset from data/demo.train.1.pt, number of examples: 10000
Step 9900, 100000; acc:  59.22; ppl:   5.49; xent:   1.70; lr: 1.00000; 4926 / 4436 tok/s;   2637 sec
Step 9950, 100000; acc:  64.17; ppl:   4.26; xent:   1.45; lr: 1.00000; 7627 / 6924 tok/s;   2650 sec
Step 10000, 100000; acc:  64.87; ppl:   4.44; xent:   1.49; lr: 1.00000; 4288 / 4225 tok/s;   2663 sec
Loading valid dataset from data/demo.valid.1.pt, number of examples: 2819
Validation perplexity: 8379.74
Validation accuracy: 16.3435
Saving checkpoint demo-model_step_10000.pt
GPU 0: for information we completed an epoch at step 10049
Loading train dataset from data/demo.train.1.pt, number of examples: 10000
Step 10050, 100000; acc:  52.43; ppl:   8.35; xent:   2.12; lr: 1.00000; 5995 / 5589 tok/s;   2683 sec
Step 10100, 100000; acc:  93.80; ppl:   1.41; xent:   0.35; lr: 1.00000; 2254 / 3393 tok/s;   2695 sec
Step 10150, 100000; acc:  52.90; ppl:   8.33; xent:   2.12; lr: 1.00000; 5719 / 5469 tok/s;   2708 sec
Step 10200, 100000; acc:  73.04; ppl:   2.95; xent:   1.08; lr: 1.00000; 5330 / 6146 tok/s;   2722 sec
GPU 0: for information we completed an epoch at step 10206
Loading train dataset from data/demo.train.1.pt, number of examples: 10000

[vince62s]

closing this, reopen it if needed

[devinbostIL]

It's running to the end without crashing now.