How to reproduce 150 TPS using FP8 + MTP=0 + BSZ=1 on H200?

[ghostplant]

So glad to see a great update of TRT-LLM which largely improves H200x8 to 150 TPS for R1. But what I get locally is just 7 TPS. What's the correct command to enjoy 150 TPS?

[juney-nvidia]

@jiahanc Hi Cyrus, I think you are the right person to answer this question? :)

cc @NVGaryJi for vis also.

[jiahanc]

Hi @ghostplant ,
The 150 TPS is with MTP = 3. We have a PR to document the reproduction steps on both Hopper and Blackwell: #3232

[karljang]

I hope the comments above addressed your question. If the problem persists in the latest release, please feel free to open a new issue. Thank you!

[ghostplant]

Hi @ghostplant , The 150 TPS is with MTP = 3. We have a PR to document the reproduction steps on both Hopper and Blackwell: #3232

Can you share the output TPS without using MTP / lookahead? This question is still not answered util now.