Unexpected latency of StarCoder when enable tensor parallel

[zhaoyang-star]

The latency of StarCoder running on 2 A100 40GB is higher than that running on 1 A100.
While, the latency of LLaMA-13B running on 2 A100 40GB is lower than that running on 1 A100 as expected.

So does MultiQueryAttenion impl in vLLM cause this?

Note: Prompt token length and output token length both are set to 1k.

[zhaoyang-star]

I found that the running requests is ~6 when max_num_seqs is set to 10 for LLaMA-13B. Meanwhile, the running requests is 10 when max_num_seqs is set to 10 for StarCoder. So the above chart about LLaMA-13B could not be used.

After I set max_num_seqs to 6 for both models and found that the latency of the two models on 2 A100 are both larger than that on 1 A100. So MultiQueryAttention may not be the reason.