CANN: Add MROPE and IMROPE support #17401
Conversation
github-actions
bot
added
testing
|
Backend CANN0: OK Verified through tests on the qwen2.5VL-7B, qwen3VL-8B, and qwen3VL-30B-A3B models. |
| acl_theta_scale_tensor.get()); | ||
|
|
||
| if (ext_factor != 0) { | ||
| // Step1.2: prepare rope_yarn_ramp, if this part updated, should update theta_scale_tensor. |
There was a problem hiding this comment.
Maybe this part, use cpu is much chaper. It will cache and calculate only once in most of cases.
noemotiovon
left a comment
noemotiovon
left a comment
There was a problem hiding this comment.
Wow! This is truly an impressive project—just understanding the code alone is really brain-intensive! The code looks fine, except that IS_VISION hasn’t been verified yet. I’d like to contribute to that part in the future as well. Finally, thank you so much for your contribution!
1. Optimize the caching logic of rope_cache_init. 2. Add support for mRoPE and i-mRoPE. Note that on Ascend 910B devices, it is necessary to disable FA in CLIP and disable NZ-format conversion. These two issues are still under investigation.
|
@ggerganov @slaren Good day! Could you please review this PR? This change make vl models available on CANN backend. Thanks. |
3 participants
Note that on Ascend 910B devices, it is necessary to disable FA
in CLIP and disable NZ-format conversion. These two issues are
still under investigation.
Make sure to read the contributing guidelines before submitting a PR