Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue training SimCLR #6

Closed
jlindsey15 opened this issue Jun 20, 2020 · 2 comments
Closed

Issue training SimCLR #6

jlindsey15 opened this issue Jun 20, 2020 · 2 comments

Comments

@jlindsey15
Copy link

jlindsey15 commented Jun 20, 2020 •
edited

When I use the SimCLR training code, my code always hangs after exactly 19 epochs and 3750 iterations. I can terminate it and restart from the previous checkpoint, but it would be nice not to have to do this. Do you have any idea why this might happen?
@XiaohangZhan
Copy link
Collaborator

Are you using pytorch-1.1 & nccl-2.4.2? If so, you may refer to: pytorch/pytorch#20630
A workaround is export NCCL_LL_THRESHOLD=0 before training.

@jlindsey15
Copy link
Author

Thanks, this fixed the problem!

XiaohangZhan pushed a commit that referenced this issue Sep 23, 2020
@xieenze
Merge pull request #6 from open-mmlab/master
c3eceef 
update
Jiahao000 pushed a commit that referenced this issue Dec 16, 2021
@fangyixiao18
[Refactor] add .github and resources folder (#6)
12ccaae 
* [Refactor] add .github folder

* [Refactor] add resources
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jlindsey15 @XiaohangZhan