Megatron-LM-v1.1.5-3D_parallelism ds_pretrain_gpt2_pipe.sh error

[zhisunyy]

Error occurred running megatron/Megatron-LM-v1.1.5-3D_parallelism/examples/ds_pretrain_gpt2_pipe.sh

finished creating GPT2 datasets ...
setting training data start iteration to 0
setting validation data start iteration to 0
done with setups ...
time (ms) | model and optimizer: 1716.01 | train/valid/test data iterators: 5358.74
training ...
[2022-07-06 11:31:24,416] [INFO] [checkpointing.py:547:forward] Activation Checkpointing Information
[2022-07-06 11:31:24,416] [INFO] [checkpointing.py:548:forward] ----Partition Activations False, CPU CHECKPOINTING False
[2022-07-06 11:31:24,416] [INFO] [checkpointing.py:551:forward] ----contiguous Memory Checkpointing False with None total layers
[2022-07-06 11:31:24,416] [INFO] [checkpointing.py:554:forward] ----Synchronization False
[2022-07-06 11:31:24,416] [INFO] [checkpointing.py:555:forward] ----Profiling time in checkpointing False
Traceback (most recent call last):
Traceback (most recent call last):
File "pretrain_gpt2.py", line 157, in
File "pretrain_gpt2.py", line 157, in
Traceback (most recent call last):
File "pretrain_gpt2.py", line 157, in
pretrain(train_valid_test_datasets_provider, model_provider, forward_step,pretrain(train_valid_test_datasets_provider, model_provider, forward_step,

File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 98, in pretrain
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 98, in pretrain
iteration = train(forward_step_func,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 481, in train
pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 98, in pretrain
Traceback (most recent call last):
iteration = train(forward_step_func,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 481, in train
File "pretrain_gpt2.py", line 157, in
iteration = train(forward_step_func,
loss_dict, skipped_iter = train_step(forward_step_func,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 481, in train
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 325, in train_step
loss_dict, skipped_iter = train_step(forward_step_func,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 325, in train_step
return train_step_pipe(model, data_iterator)
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 359, in train_step_pipe
return train_step_pipe(model, data_iterator)
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 359, in train_step_pipe
loss_dict, skipped_iter = train_step(forward_step_func,loss = model.train_batch(data_iter=data_iterator)

File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 325, in train_step
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 353, in train_batch
loss = model.train_batch(data_iter=data_iterator)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 353, in train_batch
pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 98, in pretrain
self._exec_schedule(sched)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1384, in _exec_schedule
return train_step_pipe(model, data_iterator)
self._exec_schedule(sched)
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 359, in train_step_pipe
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1384, in _exec_schedule
iteration = train(forward_step_func,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 481, in train
loss_dict, skipped_iter = train_step(forward_step_func,
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 325, in train_step
loss = model.train_batch(data_iter=data_iterator)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 353, in train_batch
self._exec_instr(**cmd.kwargs)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1034, in _exec_send_grads
return train_step_pipe(model, data_iterator)
self._exec_instr(**cmd.kwargs)
File "/root/zsf/DeepSpeedExamples-master/megatron/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 359, in train_step_pipe
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1034, in _exec_send_grads
self._exec_schedule(sched)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1384, in _exec_schedule
loss = model.train_batch(data_iter=data_iterator)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 353, in train_batch
p2p.send(inputs[1], self.prev_stage)
IndexError: tuple index out of range
p2p.send(inputs[1], self.prev_stage)
IndexError: tuple index out of range
self._exec_schedule(sched)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1384, in _exec_schedule
self._exec_instr(**cmd.kwargs)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1034, in _exec_send_grads
self._exec_instr(**cmd.kwargs)
File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1034, in _exec_send_grads
p2p.send(inputs[1], self.prev_stage)
IndexError: tuple index out of range
p2p.send(inputs[1], self.prev_stage)
IndexError: tuple index out of range

[zhisunyy]

ds_report

DeepSpeed C++/CUDA extension op report

NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.

JIT compiled ops requires ninja
ninja .................. [OKAY]

op name ................ installed .. compatible

cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
async_io ............... [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]

DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 1.11.0a0+b6df043
torch cuda version ............... 11.5
torch hip version ................ None
nvcc version ..................... 11.5
deepspeed install path ........... ['/opt/conda/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.6.5, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.5

[drcege]

Have you solved this problem？

[tjruwase]

@drcege, this is a deprecated megatron. Please use https://github.com/microsoft/Megatron-DeepSpeed

[drcege]

@tjruwase Thanks. Indeed, I was trying gpt-neox and encountered the same issue. So I search for the cause and solution.

I know microsoft's Megatron-Deepspeed, but the problem is that the Megatron version is a little old and new improvements like distributed optimizer and flash attention are not intergrated. It would be great if it is rebased onto the latest Megatron.

[tjruwase]

@drcege, you are correct that Megatron-DeepSpeed is behind latest Megatron. From the stack trace, the failure seems to be in DeepSpeed. So, the right thing to do is to open an issue with DeepSpeed using your gpt-neox failure. Would that work for you?