Why head_size must be 576 while q_head_dim of DeepSeek-V3 is only 192?

[WangNorthSea]

head_size here comes from q.sizes()[3]
But in 'modeling_deepseek.py' of DeepSeek-V3 model,
q = q.view(bsz, q_len, self.num_heads, self.q_head_dim).transpose(1, 2)
Here self.q_head_dim = config.qk_nope_head_dim + config.qk_rope_head_dim which is 128+64=192 according to 'config.json'.
How to understand this correctly?

[Kevinstone-199898]

The head size is qk_rope_head_dim+kv_lora_rank = 64+512 = 576, which is same in deepseek v2, v3 and r1. This is the absorbed version instead of the ordinary huggingface version