[BERT/PyTorch] Unable to increase the iteration count

[goyalankit]

Related to Model/Framework(s)
(e.g. GNMT/PyTorch or FasterTransformer/All) BERT/PyTorch

Describe the bug
Increasing max_steps doesn't increase the iteration count

I am trying to increase the number of iterations for BERT pytorch model so that it can train for longer duration. However changing the max_steps keeps the iteration count same as 1963. I am wondering what would the right way to increase the iteration count.

To Reproduce
Try increasing the max_steps in run_pretraining.sh for PyTorch/BERT. The iteration count remains the same.

Expected behavior
Changing max_steps should increase the iteration count.

Environment
Please provide at least:

• Container version (e.g. pytorch:19.05-py3): nvidia/pytorch:20.03-py3
• GPUs in the system: (e.g. 8x Tesla V100-SXM2-16GB): Tesla V100
• CUDA driver version (e.g. 418.67):

[forresti]

I am wondering if the new max_steps value is correctly being passed from run_pretraining.sh to run_pretraining.py.

In run_pretraining.sh, just before the line CMD="python3 -m torch.distributed.launch --nproc_per_node=$num_gpus $CMD", could you echo $CMD? And, post here with the output of echo $CMD?

https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/BERT/scripts/run_pretraining.sh#L124

[goyalankit]

Thanks for the reply:

Here's the full command that's being executed if I set max_steps to 703800

+ python3 -m torch.distributed.launch --nproc_per_node=4 /workspace/bert/DeepLearningExamples/PyTorch/LanguageModeling/BERT/run_pretraining.py --input_dir=/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_
random_seed_12345_dupe_factor_5/books_wiki_en_corpus/ --output_dir=/workspace/bert/DeepLearningExamples/PyTorch/LanguageModeling/BERT/results/checkpoints --config_file=bert_config.json --bert_model=bert-large-uncased --train_batch_size=81
92 --max_seq_length=128 --max_predictions_per_seq=20 --max_steps=703800 --warmup_proportion=0.2843 --num_steps_per_checkpoint=200 --learning_rate=6e-3 --seed=42 --fp16 --gradient_accumulation_steps=128 --allreduce_post_accumulation --allr
educe_post_accumulation_fp16 --do_train --json-summary /workspace/bert/DeepLearningExamples/PyTorch/LanguageModeling/BERT/results/dllogger.json

Even then the iteration count remains to be the same:
Iteration: 78%|███████▊ | 1535/1963

[sharathts]

1963 there refers to the number of batches available in the training shard. Training steps and losses are printed every --log_freq steps. It might take a while to see them based on this variable and gradient_accumulation_steps which is set to 128.