Concurrency / thread safety Issue with Python TrT-LLM example

[hchoi-moveworks]

Hello NVIDIA team,

Thank you developing and open sourcing this framework. 🙏

It seems like current python example for running TensorRT-LLM is not thread-safe and I wanted to bring this up.

In TensorRT guidelines, TensorRT session is thread-safe as long as new ExecutionContext is used for each thread. [1]
[1]: https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-803/best-practices/index.html#thread-safety

However, in out python TensorRT GenerationSession 2, it seems like we are using the same ExecutionContext 3 4 for a single GenerationSession.

This would make calling GenerationSession::decode function in concurrent manner unsafe.

Would there be a new GenerationSession where new ExecutionContext is created for each call to decode?
Please let me know if my understanding is incorrect.

[juney-nvidia]

@hchoi-moveworks Hi, thanks for your interest in our work!

As to your question, currently GenerationSession is not thread-safe and to ensure thread safety, different GenerationSession instances need to be created.

There are several reasons why it is designed in such a way:

• Based on our past experience, very limited Python users will use multi-thread for GPU inference, due to the impact of GIL.
• There is TensorRT-LLM C++ runtime provided which does support thread-safe inference and it is the currently suggested way for users due to performance consideration.

Thanks
June

[cody-moveworks]

@juney-nvidia Hi, thanks for your response! I have a couple of questions:

1. The GenerationSession encapsulates both the trt.ICudaEngine instance and the trt.IExecutionContext instances used for inference. My understanding is that the trt.ICudaEngine instance can be reused across threads, and that thread-safe execution can be accomplished by creating separatetrt.IExecutionContext instances for each thread. Is that correct?
2. If we create multiple GenerationSession instances, then we would also be loading duplicate trt.ICudaEngine instances, effectively loading the model weights into GPU memory multiple times. If we are using multi-threading for GPU inference in Python, it seems our best bet would be to create a separate ThreadSafeGenerationSession class that decouples the trt.ICudaEngine instance from the trt.IExecutionContext instances so that thetrt.ICudaEngine instance can be reused across multiple instances of said class. Does this approach seem reasonable?

[litaotju]

@cody-moveworks

1. it is correct.
2. In theory it can work.

Though, I'd like to understand the usage better here, due to python GIL, I think the multiple threading in python is not real multithread, so what's the benefit here would be? If there is large perf differences here, we can consider to enhance the thread safety here.

[litaotju]

I am closing this issue, since no update after my last comment. Pls re-open if you disagree. Thanks.