Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

请问可以多卡推理吗，目前单卡显存很小，跑不起来 #7

Closed
uloveqian2021 opened this issue Apr 21, 2023 · 2 comments
Closed

请问可以多卡推理吗，目前单卡显存很小，跑不起来 #7

uloveqian2021 opened this issue Apr 21, 2023 · 2 comments

Comments

@uloveqian2021
Copy link

No description provided.

@00INDEX
Copy link
Collaborator

00INDEX commented Apr 21, 2023

通过transformers的自动模型切分功能可以很方便地实现多卡推理，您可以参考 moss_cli_demo.py 中的第 24 至第 29 行以获得更多帮助。

@00INDEX 00INDEX closed this as completed Apr 22, 2023
@jacklanda
Copy link

通过transformers的自动模型切分功能可以很方便地实现多卡推理，您可以参考 moss_cli_demo.py 中的第 24 至第 29 行以获得更多帮助。

可以多卡 sft 么？目前试了下，似乎基于 accelerate 的 device_map 做 pipeline 并行的方法会造成 input 跟 下一个 block 的 layernorm 发生 different device 的错误。这个如何解决呢？谢谢~

@yangzhipeng1108 yangzhipeng1108 mentioned this issue May 11, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@00INDEX @jacklanda @uloveqian2021