Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not working with ddp_cpu #13

Closed
1 task
lvoegtlin opened this issue Jul 9, 2021 · 3 comments · Fixed by #39
Closed
1 task

Not working with ddp_cpu #13

lvoegtlin opened this issue Jul 9, 2021 · 3 comments · Fixed by #39
Assignees
Labels
bug Something isn't working If time No rush at all Pipeline The general Hydra system
Projects
Milestone

Comments

@lvoegtlin
Copy link
Contributor

lvoegtlin commented Jul 9, 2021

Describe the bug
If we want to run the framework with ddp_cpu as accelerator it wont work as it has a working directory problem.

To Reproduce
python run.py trainer.accelerator='ddp_cpu' trainer.precision=32

Expected behavior
We can use ddp_cpu to debug our system

Additional context
To avoid this problem at the moment we can just use the full path to the run.py file ($PWD/run.py).

Checklist

  • Add a warning if ddp_cpu and not presicion=32
@lvoegtlin lvoegtlin added this to To do in v0.2.X via automation Jul 9, 2021
@lvoegtlin lvoegtlin added this to the Version 1 milestone Jul 9, 2021
@lvoegtlin lvoegtlin added If time No rush at all Pipeline The general Hydra system bug Something isn't working labels Jul 9, 2021
@lvoegtlin lvoegtlin changed the title not working with ddp_cpu Not working with ddp_cpu Jul 16, 2021
@lvoegtlin
Copy link
Contributor Author

lvoegtlin commented Jul 22, 2021

When I run debug trainer with Pytorch lightning 1.3.8 I do not get an error! python run.py trainer=debug_trainer to run it. CAn you test that too @powl7 ?

@lvoegtlin lvoegtlin moved this from To do to In progress in v0.2.X Aug 30, 2021
@lvoegtlin lvoegtlin assigned lvoegtlin and powl7 and unassigned lvoegtlin Aug 30, 2021
@lvoegtlin lvoegtlin moved this from In progress to Review in progress in v0.2.X Aug 31, 2021
@lvoegtlin
Copy link
Contributor Author

We can add a user warning if he uses the ddp_cpu accelerator and the precision is not 32

@lvoegtlin lvoegtlin moved this from Review in progress to To do in v0.2.X Sep 1, 2021
@lvoegtlin lvoegtlin moved this from To do to In progress in v0.2.X Sep 1, 2021
@lvoegtlin lvoegtlin assigned lvoegtlin and unassigned powl7 Sep 1, 2021
@powl7
Copy link
Contributor

powl7 commented Sep 1, 2021

Both commands work for me. No error:
python run.py trainer.accelerator='ddp_cpu' trainer.precision=32
python run.py trainer=debug_trainer

@lvoegtlin lvoegtlin linked a pull request Sep 1, 2021 that will close this issue
@lvoegtlin lvoegtlin moved this from In progress to Review in progress in v0.2.X Sep 1, 2021
v0.2.X automation moved this from Review in progress to Done Sep 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working If time No rush at all Pipeline The general Hydra system
Projects
Development

Successfully merging a pull request may close this issue.

2 participants