Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

torch.distributed.elastic.multiprocessing.errors.ChildFailedError: #237

Open
ghost opened this issue May 19, 2023 · 25 comments
Open

torch.distributed.elastic.multiprocessing.errors.ChildFailedError: #237

ghost opened this issue May 19, 2023 · 25 comments

Comments

@ghost
Copy link

ghost commented May 19, 2023 •
edited by ghost

when i run this command:
torchrun --nproc-per-node 1 --master_port 25641 train.py --cfg-path train_configs/minigpt4_stage2_finetune.yaml

this error occurs, how can i fix it?

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 81571) of binary: /home/tiger/miniconda3/envs/minigpt4/bin/python
Traceback (most recent call last):
  File "/home/tiger/miniconda3/envs/minigpt4/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-05-19_16:43:27
  host      : n136-117-136.byted.org
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 81571)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
========================================================================================================================
@yuanlisky
Copy link 

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 198787) of binary: /home/ocr/anaconda3/envs/minigpt4/bin/python
Traceback (most recent call last):
  File "/home/ocr/anaconda3/envs/minigpt4/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
train.py FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-05-19_17:21:14
  host      : ai2
  rank      : 0 (local_rank: 0)
  exitcode  : -9 (pid: 198787)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 198787

same error

@abhijeetGithu
Copy link

The error you mentioned earlier, torch.distributed.elastic.multiprocessing.errors.ChildFailedError, typically occurs when one of the child processes launched by torchrun encounters an error and fails to execute properly.
It is difficult to pinpoint the exact cause of the error. However, here are a few possible reasons and solutions you can consider:
Resource allocation: Ensure that your system has enough resources (e.g., CPU, GPU, memory) to accommodate the requested number of child processes.

Data or code issues: Check if there are any data-related issues, such as corrupted or incompatible data. Also, review your code for any potential issues that could cause errors during training. Make sure your code is compatible with the version of PyTorch and other dependencies you are using.

Debugging the child process: Try to gather more information about the error in the child process. You can modify your code to catch and print out the specific error message or traceback for the failed child process. This will help you narrow down the issue and provide more context for troubleshooting.

Updating PyTorch and dependencies: Make sure you are using the latest version of PyTorch and related dependencies. Check for any updates or bug fixes that may address the issue you're facing. It's also a good practice to ensure that all the dependencies in your environment are compatible with each other.

Check for known issues or bugs: Search online forums, issue trackers, or the official PyTorch documentation for any known issues related to the torch.distributed.elastic.multiprocessing module. It's possible that the error you're encountering is a known issue with an existing solution or workaround.

@wujiahongPKU
Copy link

I have the same problem, I use v100 to finetune second stage using 7B

@ishitaverma
Copy link

Is there any solution for this? I am facing the same issue.

@ghost
Copy link
Author

ghost commented Jun 1, 2023 via email

@ishitaverma
Copy link

ishitaverma commented Jun 1, 2023 via email

@ghost
Copy link
Author

ghost commented Jun 5, 2023 via email

@BruceZhou95
Copy link

Is there any solution for this? I am facing the same issue.

mabey you can update lower version of torch. its work for me.

@IronSpiderMan
Copy link

i find the seem issue in hugging face, it's because of ram is not sufficient.
https://discuss.huggingface.co/t/torch-distributed-elastic-multiprocessing-errors-childfailederror/28242

@IronSpiderMan
Copy link

i find the seem issue in hugging face, it's because of ram is not sufficient. https://discuss.huggingface.co/t/torch-distributed-elastic-multiprocessing-errors-childfailederror/28242

image

@AnustupOCR
Copy link 

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 198787) of binary: /home/ocr/anaconda3/envs/minigpt4/bin/python
Traceback (most recent call last):
  File "/home/ocr/anaconda3/envs/minigpt4/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ocr/anaconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
train.py FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-05-19_17:21:14
  host      : ai2
  rank      : 0 (local_rank: 0)
  exitcode  : -9 (pid: 198787)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 198787

same error

Hi, I am having the same error while trying to Train TrOCR on multi-gpu single node setup.
My problem is not the RAM as i have 1.8TB available memory, but still i face this error.
Also i would like to point out that this particular error in the quoted reply is not as same as the original one. The exit code here is -9 as opposed to 1 in the original one.
I am also getting -9 in my case, and i am not being able to find any reason behind it.
The error is thrown randomly at the start of some epoch.
Please help me with any possible solutins if you can.

@HWH-2000
Copy link

HWH-2000 commented Aug 6, 2023

https://discuss.huggingface.co/t/torch-distributed-elastic-multiprocessing-errors-childfailederror/28242/7

Probably extending shm will solve this problem

@DengNingyuan
Copy link

I solve this problem by change the version of torch, when i use the torch2.0, i meet this question ,after i chinge the version of torch align with environment.yml ,i solve this provlem

@THUVAARAGAN
Copy link

i am using torch version 2.0.1 but i got the same torch.distributed.elastic.multiprocessing.errors.ChildFailedError
Any suggestions to this error?

@djaym7
Copy link

djaym7 commented Sep 22, 2023 

raise ChildFailedError(

torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

./run.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2023-09-22_18:11:04
host : xxx
rank : 0 (local_rank: 0)
exitcode : -11 (pid: 1775061)
error_file: <N/A>
traceback : Signal 11 (SIGSEGV) received by PID 1775061

same error

@nibrasrakib
Copy link

Anyone have any clue about this error? I am facing the same.

@joslefaure
Copy link

Decreasing the batch size worked for me

@jaouiwassim
Copy link

I had exiterror=1
I found out that I was running my code in an uncorrect environment, I had defined everything in anaconda before.
conda activate nameEnvironment
to install GPU with pytorch:
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
instead of pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
it resolved the problem for me.
Get the suitable installation command here:
https://pytorch.org/get-started/locally/

@Edenzzzz
Copy link

The error you mentioned earlier, torch.distributed.elastic.multiprocessing.errors.ChildFailedError, typically occurs when one of the child processes launched by torchrun encounters an error and fails to execute properly. It is difficult to pinpoint the exact cause of the error. However, here are a few possible reasons and solutions you can consider: Resource allocation: Ensure that your system has enough resources (e.g., CPU, GPU, memory) to accommodate the requested number of child processes.

Data or code issues: Check if there are any data-related issues, such as corrupted or incompatible data. Also, review your code for any potential issues that could cause errors during training. Make sure your code is compatible with the version of PyTorch and other dependencies you are using.

Debugging the child process: Try to gather more information about the error in the child process. You can modify your code to catch and print out the specific error message or traceback for the failed child process. This will help you narrow down the issue and provide more context for troubleshooting.

Updating PyTorch and dependencies: Make sure you are using the latest version of PyTorch and related dependencies. Check for any updates or bug fixes that may address the issue you're facing. It's also a good practice to ensure that all the dependencies in your environment are compatible with each other.

Check for known issues or bugs: Search online forums, issue trackers, or the official PyTorch documentation for any known issues related to the torch.distributed.elastic.multiprocessing module. It's possible that the error you're encountering is a known issue with an existing solution or workaround.

chatgpt

@wangsang123
Copy link

原因

I also encountered this problem. Is there any solution to this problem?

@ammaryasirnaich
Copy link

原因

I also encountered this problem. Is there any solution to this problem?

Can you share the error that you are getting on the console?
Mostly this error is caused because of running out of enough resources (GPU or memory) while in the training or inference process. It can also happen if the GPUs are not accessible in the cluster.

@adipill04
Copy link

adipill04 commented Apr 22, 2024 •
edited

It can also happen if the GPUs are not accessible in the cluster.

I know I'm not the original poster of the comment, but this is what I am getting. Any idea what exit code -6 indicates in this case? I am using 250 gb memory and 500 gb disk to run the training job so I wouldn't think it has to do with the resource allocation.

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 16) of binary: /opt/conda/bin/python
Traceback (most recent call last):
  File "/opt/conda/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.0.0', 'console_scripts', 'torchrun')())
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
===================================================
./dinov2/train/train.py FAILED
---------------------------------------------------
Failures:
[1]:
  time      : 2024-04-22_19:55:40
  host      : host.edu
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 17)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 17
---------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-04-22_19:55:40
  host      : host.edu
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 16)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 16
===================================================

@PDD0911-HCMUS
Copy link

PDD0911-HCMUS commented Apr 25, 2024 •
edited

in my case I just modified the command line to run process like this:
python -m torch.distributed.launch --nproc_per_node=2 --use_env main.py
please try add the "--use_env" before the Python file of your process
Hope can help everybody

@wxj630 wxj630 mentioned this issue May 7, 2024
Open
@tami64
Copy link

tami64 commented May 7, 2024 •
edited

I had same problem for the following sample:

To train a Swin Transformer on ImageNet from scratch, run:

python -m torch.distributed.launch --nproc_per_node --master_port 12345 main.py \
--cfg --data-path [--batch-size --output --tag ]

I solved it by removing "torch.distributed.launch --nproc_per_node --master_port 12345".

So:
To train a Swin Transformer on ImageNet from scratch, run:
python -m main.py \
--cfg --data-path [--batch-size --output --tag ]

@sennnnn
Copy link

sennnnn commented May 13, 2024

import torch before import transformers help me solve this problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests