Make TRTorch runtime thread safe

[ruoqianguo]

Signed-off-by: Ruoqian Guo ruoqiang@nvidia.com

Description

Make TRTorch runtime thread safe. We decide to just use a single stream in TRTorch runtime. If we use a single stream, then we just add a mutex for nvinfer1::IExecutionContext::enqueue. The users need to make sure that they only use a single stream, otherwise there will be an undefined behavior.

Fixes #618

Type of change

Please delete options that are not relevant and/or add your own.

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the style guidelines of this project (You can use the linters)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas and hacks
• I have made corresponding changes to the documentation
• I have added tests to verify my fix or my feature
• New and existing unit tests pass locally with my changes

[narendasan]

LGTM!