cuda driver version

[Kelsey2018]

I encountered this error, "ERROR: This container was built for NVIDIA Driver Release 535.86 or later, but version 520.56.06 was detected and compatibility mode is UNAVAILABLE."

So can TensorRT-LLM support lower driver version?

[ljayx]

Have you tried the forward compatibility mode described here?

[BasicCoder]

please reference #46

[kaiyux]

@Kelsey2018 We noticed that issue, and fixed that in the internal codebase. We plan to release an update of the main branch by the end of this week or early next week. Before that, please try the changes in #46 as a workaround, thank you.

[Kelsey2018]

Thanks a lot for replying! The performance of TensorRT-LLM is commendable; however, I am curious about the distinctive features that set it apart from the VLLM acceleration strategy. Both approaches employ paged attention, which I personally perceive as the equivalent of Paged KV Cache in TRT-LLM, along with dynamic batching to efficiently handle multiple requests.

It remains to be seen whether there will be additional documentation in the future that elucidates the unique attributes of TensorRT-LLM?

[jdemouth-nvidia]

The most fundamental technical difference is that TensorRT-LLM relies on TensorRT ; which is a graph compiler that can produce optimised kernels for your graph. As we continue to improve TensorRT, there will be less and less needs for "manual" interventions to optimise new networks (in terms of kernels as well as taking advantage of numerical optimizations like INT4, INT8 or FP8). I hope it helps a bit.

[Shixiaowei02]

This issue has been fixed in the main branch, so I will close the issue and reopen it anytime if you have new questions. Thanks!