Training problem #644
Unanswered
tiomaldy
asked this question in
General Q&A
Training problem
#644
Replies: 1 comment 3 replies
-
|
It's not normal. Something got to be NaN in your model. Try disabling mixed precision training or play with the hyper parameters. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
I going to try no disable mixed hope work this time because i waste like 4 hours for this |
Beta Was this translation helpful? Give feedback.
-
|
Can you share the config json ? |
Beta Was this translation helpful? Give feedback.
-
|
If i change the mixed to false ,i got a CUDA error memory |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Someone got this problem or its normal ?
The training stop at this point
Beta Was this translation helpful? Give feedback.
All reactions