Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

training errors #33

Closed
mycomedico opened this issue Mar 27, 2024 · 1 comment
Closed

training errors #33

mycomedico opened this issue Mar 27, 2024 · 1 comment

Comments

@mycomedico
Copy link

I get the following errors when i try to run the training script:

/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py:183: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects --local-rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions

warnings.warn(
[2024-03-27 23:53:06,101] torch.distributed.run: [WARNING]
[2024-03-27 23:53:06,101] torch.distributed.run: [WARNING] *****************************************
[2024-03-27 23:53:06,101] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-03-27 23:53:06,101] torch.distributed.run: [WARNING] *****************************************
usage: train.py [-h] -opt OPT [--launcher {none,pytorch,slurm}] [--auto_resume] [--debug]
[--local_rank LOCAL_RANK] [--force_yml FORCE_YML [FORCE_YML ...]]
usage: train.py [-h] -opt OPT [--launcher {none,pytorch,slurm}] [--auto_resume] [--debug]
[--local_rank LOCAL_RANK] [--force_yml FORCE_YML [FORCE_YML ...]]
train.py: error: unrecognized arguments: --local-rank=2
train.py: error: unrecognized arguments: --local-rank=0
usage: train.py [-h] -opt OPT [--launcher {none,pytorch,slurm}] [--auto_resume] [--debug]
[--local_rank LOCAL_RANK] [--force_yml FORCE_YML [FORCE_YML ...]]
train.py: error: unrecognized arguments: --local-rank=3
usage: train.py [-h] -opt OPT [--launcher {none,pytorch,slurm}] [--auto_resume] [--debug]
[--local_rank LOCAL_RANK] [--force_yml FORCE_YML [FORCE_YML ...]]
train.py: error: unrecognized arguments: --local-rank=1
[2024-03-27 23:53:31,246] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 2) local_rank: 0 (pid: 56905) of binary: /usr/local/bin/python3
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 198, in
main()
File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 194, in main
launch(args)
File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 179, in launch
run(args)
File "/usr/local/lib/python3.9/site-packages/torch/distributed/run.py", line 803, in run
elastic_launch(
File "/usr/local/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 135, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

basicsr/train.py FAILED

Failures:
[1]:
time : 2024-03-27_23:53:31
host : 7b4bdae11b42
rank : 1 (local_rank: 1)
exitcode : 2 (pid: 56906)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2024-03-27_23:53:31
host : 7b4bdae11b42
rank : 2 (local_rank: 2)
exitcode : 2 (pid: 56907)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2024-03-27_23:53:31
host : 7b4bdae11b42
rank : 3 (local_rank: 3)
exitcode : 2 (pid: 56908)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
time : 2024-03-27_23:53:31
host : 7b4bdae11b42
rank : 0 (local_rank: 0)
exitcode : 2 (pid: 56905)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

@piddnad
Copy link
Owner

piddnad commented Jul 24, 2024

It appears that the issue was due to the local-rank parameter not being correctly parsed, which has been fixed in commit 74293b6. Will close this issue.

@piddnad piddnad closed this as completed Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants