New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[deepspeed] 0VERFLOW! Rank 0 Skipping step. Attempted os scale: 65536！！！ #320

Open

PHI6kai opened this issue Nov 21, 2023 · 2 comments

PHI6kai commented Nov 21, 2023

不管训大的训练集还是小的都会显示overflow然后全为0，是内存问题还是显存问题呢，用的A100-80G
@Sleepychord

Member

1049451037 commented Nov 22, 2023

这是deepspeed的scaling机制，可以防止overflow。

bandianxuediao commented Apr 10, 2024

这是deepspeed的scaling机制，可以防止overflow。

也就是说这个是正常的么？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment