[feat]: Support vllm calibrate for hy_v3.#309
Merged
Conversation
C-KRSW
force-pushed
the
hyv3_vllm_calib
branch
2 times, most recently
from
181bf35 to
258f49a
Compare
yghstill
reviewed
May 20, 2026
| print("---------KV-cache hooks skipped (kv_granularity=none)---------") | ||
|
|
||
| elif kv_granularity == "per-tensor": | ||
| kvcache_layers = _find_layers(model, layers=[Attention]) |
Collaborator
There was a problem hiding this comment.
代码风格保持一致，per-head调用setup_kvcache_perhead_hooks，per-tensor也这样调用hook函数
| ) | ||
|
|
||
|
|
||
| class KVScaleSearcher: |
Collaborator
There was a problem hiding this comment.
kv-cache scale search的逻辑再拆分到新的py文件，当前文件代码行数太多
|
|
||
| int(parts[expert_idx + 1]) # 尝试转换为整数 | ||
|
|
||
| # # 检查 expert 之前是否存在 gate/up/down |
| from safetensors import safe_open | ||
| from safetensors.torch import save_file | ||
|
|
||
| json_path = "/apdcephfs_gy7/share_303171455/linchuanxie/code_0403_kvcache_tensor/kv_cache_tuned_scales.json" # noqa: E501 |
| @@ -0,0 +1,452 @@ | |||
| # Copyright 2025 Tencent Inc. All Rights Reserved. | |||
Collaborator
There was a problem hiding this comment.
关于kvcache量化相关的在tools下新建一个文件夹
Collaborator
Author
There was a problem hiding this comment.
暂时将p scale的文件删掉，并将其余关于kvcache量化相关的文件移动到新目录下.
yghstill
approved these changes
May 21, 2026
cctvqxj
pushed a commit
to cctvqxj/AngelSlim
that referenced
this pull request
Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.