Deepspeed: Tensor Parallelism with 2 GPUs Falls Short of Single GPU Performance

[thri5ha]

When inferencing Llama7b in A100 GPU 40GB,

FP16 datatype - Single GPU performs better than TP=2

FP32 datatype - TP = 2 performs better, however the overall throughput is reduced.

1. Is there any reason on why in F16 datatype, the single GPU outperforms TP-2 ?
2. Can you please let me know, if there are other methods to improve the inference throughput ?