-
Notifications
You must be signed in to change notification settings - Fork 25.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NaN in GPT NeoX model (generation) #17452
Comments
cc @zphang do you have an idea of what might be happening there? Also happening here: huggingface/accelerate#404 |
Also cc @sgugger as it's leveraging the auto map. |
Seems to be a float16 overflow problem after applying
Although after fixing this NaN problem, the generation is still not working correctly. |
Thanks @zomux, this works. I tried something similar (casting to FP32 and back) which gave the same error:
I figured out that when setting use_cache=False generation runs without errors.
|
Ah, I thought I'd removed all the RuntimeErrors, I've submitted a PR here: #17563 As for the NaN-ing, I've not figured that out either. I found (in very ad-hoc testing) that the einsum appears to be more stable than the original approach, but it looks like it hasn't fully solved the issue. |
Hello @zphang,
It seems to work fine:
|
@zphang Is there a clear release version where the issue is resolved? |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
System Info
Who can help?
@LysandreJik (NeoX)
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Script to run:
Script is crashing with the traceback:
The problem was also mentioned here:
#15642 (comment)
#15642 (comment)
The problem seems to be that
torch.einsum
returnsinf
(fp16 overflow) which leads tonan
when calculating the softmax.Expected behavior
The text was updated successfully, but these errors were encountered: