Access violation when repeatedly creating/destroying inference session for TensorRT Execution Provider

[nietras]

Describe the bug

Access violation error being thrown during the engine building phase when using TensorRT ExecutionProvider when repeatedly creating and deallocating the inference session.

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 x64
ONNX Runtime installed from (source or binary): binary
ONNX Runtime version: 1.19.2 AND 1.21.0
Python version: N/A (using C# wrapper)
Visual Studio version (if applicable): 2022
CUDA version:11.8.0 AND 12.8.1
cuDNN version: 8.9.7.29 AND 9.8.0.87
TensorRT version: 10.5.0.18 AND 10.8.0.43
GPU model and memory: RTX 3080

To Reproduce

Unfortunately, we have not been able to reproduce the access violation isolated, but in our use case we are simply creating a model from onnx with TensorRT Execution Provider, run for a while, then destroy it. Run without. Then create a new. And then after some N repeating that for unknown reasons the access violation occurs. Only need to do it a handful of times. Always happens. Note model works fine usually and has for a few years. Basic CNN.

Screenshots

Code

https://github.com/microsoft/onnxruntime/blob/v1.19.2/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc#L2258

Additional context

As can be seen from screenshot we have been trying hard to debug this, but we have hit a dead end, we do not understand why this occurs. Hence, this issue, in the hope we can get help finding the issue and remedying. Understandably, this is a bit hard given we have been unable to isolate code in question that actually reproduces despite trying this, we are unsure why this is not possible, since we do not do anything that different in our app itself.

Similar issue is Access violation when using TensorRT ExecutionProvider on multiple GPU #7322 but no solution there for us it seems.

[jywu-msft]

do you know if this issue reproduces with native TensorRT (without OnnxRuntime)? i.e. using trt_exec to build the engine? (assuming the full graph is supported by TensorRT)

[nietras]

@jywu-msft it does not happen with TensorRT/trt_exec (at least not seen it). Stack trace also show error happens in ort trt provider.

[jywu-msft]

@jywu-msft it does not happen with TensorRT/trt_exec (at least not seen it). Stack trace also show error happens in ort trt provider.

The access violation happens in createNetworkV2 though, right? (TensorRT code)
Btw, the trace seems to indicate TensorRT 10.2.0.19 , but you mentioned you tested with TensorRT 10.5 and 10.8 ?
is there a model you can share for repro purposes? we haven't seen this type of failure in our automated testing.
@chilo-ms

[nietras]

The access violation happens in createNetworkV2 though, right? (TensorRT code)

Not as far as I can tell, my interpretation here is this happens when trying to "call" trt_builder->createNetworkV2 (which is inlined), but trt_builder is null (rax). Hence, the problem is trt_builder not being created, perhaps also no checks for this being null etc.

onnxruntime/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc

Lines 2252 to 2258 in ffceed9

	auto trt_builder = GetBuilder(trt_logger);
	auto network_flags = 0;
	#if NV_TENSORRT_MAJOR > 8
	network_flags |= fp16_enable_ || int8_enable_ ? 0 : 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kSTRONGLY_TYPED);
	#endif
	network_flags |= 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
	auto trt_network = std::unique_ptr<nvinfer1::INetworkDefinition>(trt_builder->createNetworkV2(network_flags));

[nietras]

Btw, the trace seems to indicate TensorRT 10.2.0.19 , but you mentioned you tested with TensorRT 10.5 and 10.8 ?

The version in trace is the version ORT TRT was build against, not my code. But yes we have tested 10.2.0.19 too.

We cannot share a model unfortunately.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[nietras]

Not resolved.

[jywu-msft]

@chilo-ms

[jywu-msft]

@gedoensmax , @ankan-ban have you ever seen this behavior in your testing?