Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

训练问题 #34

Closed
ZZFanya-DWR opened this issue Aug 6, 2021 · 2 comments
Closed

训练问题 #34

ZZFanya-DWR opened this issue Aug 6, 2021 · 2 comments

Comments

@ZZFanya-DWR
Copy link

作者，您好，我想问下，为什么我在本地训练都是正常的，然后在集群训练loss总是NAN，我尝试从新建环境，最终训练，始终保持本地和集群环境一直，可是最终还是本地正常，集群异常，请问怎么回事，谢谢
@xinntao
Copy link
Member

xinntao commented Aug 6, 2021

这个我也不清楚了， 你需要自己debug一下了。 我没有遇到过这个情况~

@xinntao xinntao closed this as completed Aug 9, 2021
@zhoumenghan
Copy link

作者，您好，我想问下，为什么我在本地训练都是正常的，然后在集群训练loss总是NAN，我尝试从新建环境，最终训练，始终保持本地和集群环境一直，可是最终还是本地正常，集群异常，请问怎么回事，谢谢

请问您后来是怎么解决这个问题的，我也遇到了这个情况。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@xinntao @zhoumenghan @ZZFanya-DWR