Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] CUDA Out of Memory when eval model. #133

Open
5 tasks done
Crystalxd opened this issue Sep 12, 2023 · 5 comments
Open
5 tasks done

[BUG] CUDA Out of Memory when eval model. #133

Crystalxd opened this issue Sep 12, 2023 · 5 comments
Labels
bug Something isn't working

Comments

@Crystalxd
Copy link

Required prerequisites

System information

conda environment
torch=2.0.1
transformers=4.29.2
...

Problem description

I used A100(80G) to run the evaluate_zh.py script for evaluating baichuan model, but it occupied abundant GPU memory up to overflow. Then I found the model loaded without eval mode, meanwhile, it inferred without no_grad.

Reproducible example code

The Python snippets:

[https://github.com/baichuan-inc/Baichuan-7B/blob/6f3ef4633a90c2d8a3e0763d0dec1b8dc11588f5/evaluation/evaluate_zh.py#L97C13-L97C13](url)
self.model = model.eval()

https://github.com/baichuan-inc/Baichuan-7B/blob/6f3ef4633a90c2d8a3e0763d0dec1b8dc11588f5/evaluation/evaluate_zh.py#L103
Add on this line:
@torch.inference_mode()

Command lines:

Extra dependencies:


Steps to reproduce:

Traceback

No response

Expected behavior

No response

Additional context

No response

Checklist

  • I have provided all relevant and necessary information above.
  • I have chosen a suitable title for this issue.
@Crystalxd Crystalxd added the bug Something isn't working label Sep 12, 2023
@Guanze-Chen
Copy link

Thank you. It works!!!

@ICanFlyGFC
Copy link

Thanks!

@Guanze-Chen
Copy link

Guanze-Chen commented Dec 7, 2023 via email

@Young-X
Copy link

Young-X commented Jul 24, 2024

我在训练模型过程中,脚本默认使用gpu0,怎么调换到gpu1上面?

@Guanze-Chen
Copy link

Guanze-Chen commented Jul 24, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants