Running on SLURM cluster with multiple GPUs gets stuck at dataset creation step

[simonblauth]

I am having trouble using this repository. Any ideas what to do or try?

Description

I want to run my submission on a SLURM cluster without docker.
For now I was trying to run the baselines to test the setup.
Running with 1 gpu works fine but the process gets stuck when using multiple GPUs.

running with 1 gpu

command used:

torchrun --standalone --nnodes=1 --nproc_per_node=1 submission_runner.py --framework=pytorch --workload=mnist --experiment_dir=experiments --experiment_name=baseline_adamw --submission_path=baselines/adamw/pytorch/submission.py --tuning_search_space=baselines/adamw/tuning_search_space.json --torch_compile=False --overwrite

this works fine and runs the submission as expected. Abbreviated console output:

master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
I1114 19:32:40.648843 140249311245440 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 0
I1114 19:32:40.649153 140249311245440 distributed_c10d.py:476] Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
I1114 19:32:41.436739 140249311245440 logger_utils.py:76] Creating experiment directory at experiments/baseline_adamw/mnist_pytorch.
I1114 19:32:41.453138 140249311245440 submission_runner.py:526] Using RNG seed 1762019175
I1114 19:32:41.454235 140249311245440 submission_runner.py:535] --- Tuning run 1/1 ---
I1114 19:32:41.454341 140249311245440 submission_runner.py:540] Creating tuning directory at experiments/baseline_adamw/mnist_pytorch/trial_1.
I1114 19:32:41.455712 140249311245440 logger_utils.py:92] Saving hparams to experiments/baseline_adamw/mnist_pytorch/trial_1/hparams.json.
I1114 19:32:41.458003 140249311245440 submission_runner.py:205] Initializing dataset.
I1114 19:32:41.458118 140249311245440 submission_runner.py:212] Initializing model.
I1114 19:32:42.231244 140249311245440 submission_runner.py:243] Initializing optimizer.
I1114 19:32:42.231692 140249311245440 submission_runner.py:250] Initializing metrics bundle.
I1114 19:32:42.231785 140249311245440 submission_runner.py:268] Initializing checkpoint and logger.
I1114 19:32:42.233011 140249311245440 submission_runner.py:288] Saving meta data to experiments/baseline_adamw/mnist_pytorch/trial_1/meta_data_0.json.
I1114 19:32:42.460128 140249311245440 submission_runner.py:292] Saving flags to experiments/baseline_adamw/mnist_pytorch/trial_1/flags_0.json.
I1114 19:32:42.499783 140249311245440 submission_runner.py:302] Starting training loop.
I1114 19:32:42.512056 140249311245440 dataset_info.py:578] Load dataset info from /home/blauths/data/mnist/3.0.1
I1114 19:32:42.515055 140249311245440 dataset_info.py:669] Fields info.[citation, splits, supervised_keys, module_name] from disk and from code do not match. Keeping the one from code.
I1114 19:32:42.515373 140249311245440 dataset_builder.py:528] Reusing dataset mnist (/home/blauths/data/mnist/3.0.1)
I1114 19:32:42.597422 140249311245440 logging_logger.py:49] Constructing tf.data.Dataset mnist for split train[:50000], from /home/blauths/data/mnist/3.0.1
I1114 19:32:44.257822 140230997542464 logging_writer.py:48] [0] global_step=0, grad_norm=208.853928, loss=18.818687
I1114 19:32:44.281792 140249311245440 submission.py:120] 0) loss = 18.819, grad_norm = 208.854
I1114 19:32:44.467674 140249311245440 spec.py:321] Evaluating on the training split.
I1114 19:32:44.470147 140249311245440 dataset_info.py:578] Load dataset info from /home/blauths/data/mnist/3.0.1
I1114 19:32:44.473052 140249311245440 dataset_info.py:669] Fields info.[citation, splits, supervised_keys, module_name] from disk and from code do not match. Keeping the one from code.
I1114 19:32:44.473357 140249311245440 dataset_builder.py:528] Reusing dataset mnist (/home/blauths/data/mnist/3.0.1)
I1114 19:32:44.522919 140249311245440 logging_logger.py:49] Constructing tf.data.Dataset mnist for split train[:50000], from /home/blauths/data/mnist/3.0.1
I1114 19:32:49.728950 140249311245440 spec.py:333] Evaluating on the validation split.
I1114 19:32:49.731426 140249311245440 dataset_info.py:578] Load dataset info from /home/blauths/data/mnist/3.0.1
I1114 19:32:49.734574 140249311245440 dataset_info.py:669] Fields info.[citation, splits, supervised_keys, module_name] from disk and from code do not match. Keeping the one from code.
I1114 19:32:49.734902 140249311245440 dataset_builder.py:528] Reusing dataset mnist (/home/blauths/data/mnist/3.0.1)
I1114 19:32:49.786858 140249311245440 logging_logger.py:49] Constructing tf.data.Dataset mnist for split train[50000:], from /home/blauths/data/mnist/3.0.1
I1114 19:32:50.280914 140249311245440 spec.py:349] Evaluating on the test split.
I1114 19:32:50.283240 140249311245440 dataset_info.py:578] Load dataset info from /home/blauths/data/mnist/3.0.1
I1114 19:32:50.286474 140249311245440 dataset_info.py:669] Fields info.[citation, splits, supervised_keys, module_name] from disk and from code do not match. Keeping the one from code.
I1114 19:32:50.286855 140249311245440 dataset_builder.py:528] Reusing dataset mnist (/home/blauths/data/mnist/3.0.1)
I1114 19:32:50.341554 140249311245440 logging_logger.py:49] Constructing tf.data.Dataset mnist for split test, from /home/blauths/data/mnist/3.0.1
I1114 19:32:50.871682 140249311245440 submission_runner.py:396] Time since start: 8.37s, 	Step: 1, 	{'train/accuracy': 0.0499, 'train/loss': 1.18980361328125, 'validation/accuracy': 0.0518, 'validation/loss': 1.19333125, 'validation/num_examples': 10000, 'test/accuracy': 0.0552, 'test/loss': 1.188576171875, 'test/num_examples': 10000, 'score': 1.7838444709777832, 'total_duration': 8.372186660766602, 'accumulated_submission_time': 1.7838444709777832, 'accumulated_eval_time': 6.404211759567261, 'accumulated_logging_time': 0}
.
.
.
I1114 19:33:11.261379 140249311245440 spec.py:321] Evaluating on the training split.
I1114 19:33:11.271509 140249311245440 spec.py:333] Evaluating on the validation split.
I1114 19:33:11.281817 140249311245440 spec.py:349] Evaluating on the test split.
.
.
.
I1114 19:33:55.048449 140249311245440 submission_runner.py:569] Timing: 60.00460410118103
I1114 19:33:55.048520 140249311245440 submission_runner.py:571] Total number of evals: 7
I1114 19:33:55.048587 140249311245440 submission_runner.py:572] ====================
I1114 19:33:55.048690 140249311245440 submission_runner.py:651] Final mnist score: 60.00460410118103

running with 2 gpus

command used:

torchrun --redirects 1:0 --standalone --nnodes=1 --nproc_per_node=2 submission_runner.py --framework=pytorch --workload=mnist --experiment_dir=experiments --experiment_name=baseline_adamw --submission_path=baselines/adamw/pytorch/submission.py --tuning_search_space=baselines/adamw/tuning_search_space.json --torch_compile=False --overwrite

this gets stuck at the dataset creation step until timeout.
See the following console output for where it stops:

master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
I1114 19:39:46.754390 140163014198400 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 0
I1114 19:39:46.754344 139726293963904 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 1
I1114 19:39:46.754640 140163014198400 distributed_c10d.py:476] Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
I1114 19:39:46.754694 139726293963904 distributed_c10d.py:476] Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
I1114 19:39:47.555879 140163014198400 logger_utils.py:61] Removing existing experiment directory experiments/baseline_adamw/mnist_pytorch because --overwrite was set.
I1114 19:39:47.560014 140163014198400 logger_utils.py:76] Creating experiment directory at experiments/baseline_adamw/mnist_pytorch.
I1114 19:39:47.574892 140163014198400 submission_runner.py:526] Using RNG seed 545346501
I1114 19:39:47.575957 140163014198400 submission_runner.py:535] --- Tuning run 1/1 ---
I1114 19:39:47.576064 140163014198400 submission_runner.py:540] Creating tuning directory at experiments/baseline_adamw/mnist_pytorch/trial_1.
I1114 19:39:47.577447 140163014198400 logger_utils.py:92] Saving hparams to experiments/baseline_adamw/mnist_pytorch/trial_1/hparams.json.
I1114 19:39:47.579491 140163014198400 submission_runner.py:205] Initializing dataset.
I1114 19:39:47.579612 140163014198400 submission_runner.py:212] Initializing model.
I1114 19:39:48.925061 140163014198400 submission_runner.py:243] Initializing optimizer.
I1114 19:39:48.925512 140163014198400 submission_runner.py:250] Initializing metrics bundle.
I1114 19:39:48.925613 140163014198400 submission_runner.py:268] Initializing checkpoint and logger.
I1114 19:39:48.926866 140163014198400 submission_runner.py:288] Saving meta data to experiments/baseline_adamw/mnist_pytorch/trial_1/meta_data_0.json.
I1114 19:39:49.135014 140163014198400 submission_runner.py:292] Saving flags to experiments/baseline_adamw/mnist_pytorch/trial_1/flags_0.json.
I1114 19:39:49.175033 140163014198400 submission_runner.py:302] Starting training loop.
I1114 19:39:49.187256 140163014198400 dataset_info.py:578] Load dataset info from /home/blauths/data/mnist/3.0.1
I1114 19:39:49.190353 140163014198400 dataset_info.py:669] Fields info.[citation, splits, supervised_keys, module_name] from disk and from code do not match. Keeping the one from code.
I1114 19:39:49.190659 140163014198400 dataset_builder.py:528] Reusing dataset mnist (/home/blauths/data/mnist/3.0.1)
I1114 19:39:49.270506 140163014198400 logging_logger.py:49] Constructing tf.data.Dataset mnist for split train[:50000], from /home/blauths/data/mnist/3.0.1

torch compile

The behavior does not change when enabling/disabling torch.compile.

using singularity/apptainer

I built the singularity container from the Docker image.
Running the above commands in the singularity container exhibits the same behaviour.
It still gets stuck with two gpus and works fine with one.

Steps to Reproduce

• clone repository
• setup conda env as described in README.md
• follow installation for pytorch version
• run above command on machine with multiple GPUs

[priyakasimbeg]

Thanks for filing this issue.
A few questions:

1. Does this also happen on the other workloads?
2. Which version of CUDA and which driver are you using? Could you paste the output of nvidia-smi here?

[simonblauth]

Thanks for the fast reply!
here is the nvidia-smi output:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     On  | 00000000:43:00.0 Off |                  N/A |
|  0%   33C    P8               1W / 250W |      1MiB / 11264MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 2080 Ti     On  | 00000000:C3:00.0 Off |                  N/A |
|  0%   33C    P8              19W / 250W |      1MiB / 11264MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

I have not yet tested the other workloads as I first wanted to test if it works in general. I will check if other workloads run.

[simonblauth]

So I tested two different workloads and ogbg works fine.
When running wmt I get the following error:

cpu-bind=MASK - dlcgpu21, task  0  0 [1891]: mask 0xffffffffffffffff set
master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
I1115 18:28:21.402525 140029388997696 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 5
I1115 18:28:21.402787 139813540795456 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 4
I1115 18:28:21.405364 140393551533120 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 3
I1115 18:28:21.408071 140705261016128 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 1
I1115 18:28:21.409652 140334553838656 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 6
I1115 18:28:21.410261 140514822829120 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 7
I1115 18:28:22.396781 139767885591616 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 2
I1115 18:28:22.407059 140581994722368 distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 0
I1115 18:28:22.407266 139767885591616 distributed_c10d.py:476] Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:22.407495 140581994722368 distributed_c10d.py:476] Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:22.412158 140029388997696 distributed_c10d.py:476] Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:22.412218 139813540795456 distributed_c10d.py:476] Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:22.412699 140393551533120 distributed_c10d.py:476] Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:22.414368 140705261016128 distributed_c10d.py:476] Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:22.416082 140334553838656 distributed_c10d.py:476] Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:22.416110 140514822829120 distributed_c10d.py:476] Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
I1115 18:28:25.167300 140581994722368 logger_utils.py:61] Removing existing experiment directory experiments/baseline_adamw/wmt_pytorch because --overwrite was set.
I1115 18:28:25.171687 140581994722368 logger_utils.py:76] Creating experiment directory at experiments/baseline_adamw/wmt_pytorch.
I1115 18:28:25.193192 140581994722368 submission_runner.py:526] Using RNG seed 2712291374
I1115 18:28:25.193775 140581994722368 submission_runner.py:535] --- Tuning run 1/1 ---
I1115 18:28:25.193826 140581994722368 submission_runner.py:540] Creating tuning directory at experiments/baseline_adamw/wmt_pytorch/trial_1.
I1115 18:28:25.196289 140581994722368 logger_utils.py:92] Saving hparams to experiments/baseline_adamw/wmt_pytorch/trial_1/hparams.json.
I1115 18:28:25.197938 140581994722368 submission_runner.py:205] Initializing dataset.
I1115 18:28:25.197991 140581994722368 submission_runner.py:212] Initializing model.
I1115 18:28:27.599292 140581994722368 submission_runner.py:243] Initializing optimizer.
I1115 18:28:27.599922 140581994722368 submission_runner.py:250] Initializing metrics bundle.
I1115 18:28:27.599967 140581994722368 submission_runner.py:268] Initializing checkpoint and logger.
I1115 18:28:27.600616 140581994722368 submission_runner.py:288] Saving meta data to experiments/baseline_adamw/wmt_pytorch/trial_1/meta_data_0.json.
I1115 18:28:27.600737 140581994722368 logger_utils.py:257] Unable to record workload.train_mean information. Continuing without it.
I1115 18:28:27.600773 140581994722368 logger_utils.py:257] Unable to record workload.train_stddev information. Continuing without it.
I1115 18:28:27.890596 140581994722368 submission_runner.py:292] Saving flags to experiments/baseline_adamw/wmt_pytorch/trial_1/flags_0.json.
I1115 18:28:28.077461 140581994722368 submission_runner.py:302] Starting training loop.
I1115 18:28:28.087835 140581994722368 dataset_info.py:578] Load dataset info from /home/blauths/data/wmt/wmt17_translate/de-en/1.0.0
I1115 18:28:28.138467 140581994722368 logging_logger.py:49] Constructing tf.data.Dataset wmt17_translate for split train, from /home/blauths/data/wmt/wmt17_translate/de-en/1.0.0
Traceback (most recent call last):
  File "/home/blauths/algorithmic-efficiency/submission_runner.py", line 667, in <module>
    app.run(main)
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/blauths/algorithmic-efficiency/submission_runner.py", line 635, in main
    score = score_submission_on_workload(
  File "/home/blauths/algorithmic-efficiency/submission_runner.py", line 552, in score_submission_on_workload
    timing, metrics = train_once(workload, global_batch_size,
  File "/home/blauths/algorithmic-efficiency/submission_runner.py", line 324, in train_once
    optimizer_state, model_params, model_state = update_params(
  File "/home/blauths/algorithmic-efficiency/baselines/adamw/pytorch/submission.py", line 74, in update_params
    logits_batch, new_model_state = workload.model_fn(
  File "/home/blauths/algorithmic-efficiency/algorithmic_efficiency/workloads/wmt/wmt_pytorch/workload.py", line 212, in model_fn
    logits_batch = model(
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1156, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/blauths/algorithmic-efficiency/algorithmic_efficiency/workloads/wmt/wmt_pytorch/models.py", line 170, in forward
    memory = self.encoder(
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/blauths/algorithmic-efficiency/algorithmic_efficiency/workloads/wmt/wmt_pytorch/models.py", line 291, in forward
    memory = self.encoder(src, mask=src_mask)
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/blauths/algorithmic-efficiency/algorithmic_efficiency/workloads/wmt/wmt_pytorch/models.py", line 245, in forward
    output = mod(
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/blauths/algorithmic-efficiency/algorithmic_efficiency/workloads/wmt/wmt_pytorch/models.py", line 505, in forward
    x = x + self._sa_block(self.norm1(x), src_mask)
  File "/home/blauths/algorithmic-efficiency/algorithmic_efficiency/workloads/wmt/wmt_pytorch/models.py", line 515, in _sa_block
    x, _ = self.self_attn(x, attn_mask=attn_mask)
  File "/home/blauths/miniconda3/envs/algoperf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/blauths/algorithmic-efficiency/algorithmic_efficiency/workloads/wmt/wmt_pytorch/models.py", line 952, in forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

The error occures at the same point where mnist workload got stuck.

[priyakasimbeg]

It seems like for some reason the second GPU isn't found. Could you also print out CUDA_VISIBLE_DEVICES or in the submission runner log torch.cuda.device_count()?

@msaroufim have you seen this type of issue before?

[priyakasimbeg]

I was able to reproduce both of these symptoms, the MNIST bug should be fixed now w PR/584.
For the CUDA error: invalid device ordinal issue; this happened when the CUDA visible devices didn't match the torch run commands. Make sure you have both devices visible e.g. by running export CUDA_VISIBLE_DEVICES=0,1.

Closing this for now. Feel free to reopen if you have any related issues still.