[Performance]: is the performance of the PyTorch backend on par with the previous engine compilation setup?

[JoJoLev]

Proposal to improve performance

I do not think I really understand the move to PyTorch backend, it is said to be the default now. Does this mean the engine compilation is going away?
Also, does the torch backend have similar performance to the previous engine building steps?

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

System Information:

• OS:
• Python version:
• CUDA version:
• GPU model(s):
• Driver version:
• TensorRT version:
• PyTorch version:
• TensorRT-LLM version:

Detailed output:

Paste the output of the above commands here

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[akkolyth]

+1

[karljang]

@kaiyux , could you take a time to review this question?

[videodanchik]

+1

[karljang]

Sharing my understanding: the PyTorch backend is the core architecture for TensorRT-LLM 1.0 and beyond, so our focus will be on investing in and improving that path. We won’t be adding new features to the TensorRT backend, though it will continue to function as-is.
Regarding the performance, we believe the PyTorch backend has achieved comparable performance. However, if you notice anything unusual or off, please feel free to open a ticket so we can investigate further.

[github-actions]

Issue has not received an update in over 14 days. Adding stale label.

[riZZZhik]

+1