-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhancing Ascend 910A Training Efficiency in LlamaFactory with NPU #3584
Conversation
…anced, leveraging the full computational power of the NPU (Neural Processing Unit) and the capabilities of torch_npu, a PyTorch library optimized for NPUs. This improvement has resulted in a remarkable tenfold increase in efficiency.
We should first check if the torch_npu package is available, such as LLaMA-Factory/src/llmtuner/chat/vllm_engine.py Lines 11 to 14 in d6ca785
|
Alright, so it looks like VLLM doesn't support Ascend. No worries, I'll just tweak the code a bit and see if I can get it working. |
Yeah, I think updating the docs for now and letting the devs figure out how to handle it could be a good way to go. Just make it clear in the documentation that VLLM and Ascend aren't playing nicely together at the moment. That way, the developers can see the issue straight away and decide for themselves how they want to tackle it, based on what works best for their specific project. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm... If you want to use LLaMA-Factory on Ascend910A, this is the modification method I would recommend.
Co-authored-by: Huazhong Ji <hzji210@gmail.com>
Co-authored-by: Huazhong Ji <hzji210@gmail.com>
It now works, LGTM |
What does this PR do?
The training efficiency of the Ascend 910A has been significantly enhanced by leveraging the full computational power of the NPU and the capabilities of torch_npu. This enhancement has led to a remarkable tenfold increase in training efficiency.