New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PyTorch版本对推理性能的影响具体是在哪些方面？ #22

Closed

mockyd opened this issue Oct 27, 2023 · 1 comment

mockyd commented Oct 27, 2023

萌新~请教各位大佬，我看到文档中说需要torch2.0以上达到最佳推理性能，请问是体现在速度方面吗？会不会影响模型的推理效果呢？
谢谢大佬们！

Member

duzx16 commented Oct 28, 2023

主要是 PyTorch 2.0 以后的版本才有 Flash Attention实现，会影响 attention 的速度和显存占用，不会影响结果。

duzx16 closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment