"trt_cuda_graph_enable" bug in tensorrt.

[hy846130226]

Describe the issue

I'm using onnx-tensorrt.

When I enable the trt_cuda_graph_enable like this:

Subsequently, no matter how many images of data I pass for inference, what I get is always the result of the first image.

The following is my infer code:

the “input” and “output temp” is reusable.

To reproduce

1. use onnx-tensorrt 1.16.3/1.17.0/1.17.1
2. infer different image

Urgency

No response

Platform

Windows

OS Version

WIN10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.17.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

No response

[hy846130226]

If I disable the trt_cuda_graph_enable, I could get every image correct results.

[tianleiwu]

Make sure you use IO/Binding to bind input tensors in GPU memory. During inference, copy input to same address (input shape shall be the same) of the input used in the first inference run.

You can get some idea from corresponding python code:

onnxruntime/onnxruntime/python/tools/transformers/io_binding_helper.py

Lines 212 to 307 in 4a196d1

	class CudaSession:
	"""Inference Session with IO Binding for ONNX Runtime CUDA or TensorRT provider"""
	def __init__(self, ort_session: InferenceSession, device: torch.device, enable_cuda_graph=False):
	self.ort_session = ort_session
	self.input_names = [input.name for input in self.ort_session.get_inputs()]
	self.output_names = [output.name for output in self.ort_session.get_outputs()]
	self.io_name_to_numpy_type = TypeHelper.get_io_numpy_type_map(self.ort_session)
	self.io_binding = self.ort_session.io_binding()
	self.enable_cuda_graph = enable_cuda_graph
	self.input_tensors = OrderedDict()
	self.output_tensors = OrderedDict()
	self.device = device
	def __del__(self):
	del self.input_tensors
	del self.output_tensors
	del self.io_binding
	def allocate_buffers(self, shape_dict: Dict[str, Union[Tuple[int], List[int]]]):
	"""Allocate tensors for I/O Binding"""
	if self.enable_cuda_graph:
	for name, shape in shape_dict.items():
	if name in self.input_names:
	# Reuse allocated buffer when the shape is same
	if name in self.input_tensors:
	if tuple(self.input_tensors[name].shape) == tuple(shape):
	continue
	raise RuntimeError("Expect static input shape for cuda graph")
	numpy_dtype = self.io_name_to_numpy_type[name]
	tensor = torch.empty(tuple(shape), dtype=TypeHelper.numpy_type_to_torch_type(numpy_dtype)).to(
	device=self.device
	)
	self.input_tensors[name] = tensor
	self.io_binding.bind_input(
	name,
	tensor.device.type,
	tensor.device.index,
	numpy_dtype,
	list(tensor.size()),
	tensor.data_ptr(),
	)
	for name, shape in shape_dict.items():
	if name in self.output_names:
	# Reuse allocated buffer when the shape is same
	if name in self.output_tensors and tuple(self.output_tensors[name].shape) == tuple(shape):
	continue
	numpy_dtype = self.io_name_to_numpy_type[name]
	tensor = torch.empty(tuple(shape), dtype=TypeHelper.numpy_type_to_torch_type(numpy_dtype)).to(
	device=self.device
	)
	self.output_tensors[name] = tensor
	self.io_binding.bind_output(
	name,
	tensor.device.type,
	tensor.device.index,
	numpy_dtype,
	list(tensor.size()),
	tensor.data_ptr(),
	)
	def infer(self, feed_dict: Dict[str, torch.Tensor], run_options: RunOptions = None, synchronize: bool = False):
	"""Bind input tensors and run inference"""
	for name, tensor in feed_dict.items():
	assert isinstance(tensor, torch.Tensor) and tensor.is_contiguous()
	if name in self.input_names:
	if self.enable_cuda_graph:
	assert self.input_tensors[name].nelement() == tensor.nelement()
	assert self.input_tensors[name].dtype == tensor.dtype
	assert tensor.device.type == "cuda"
	self.input_tensors[name].copy_(tensor)
	else:
	self.io_binding.bind_input(
	name,
	tensor.device.type,
	tensor.device.index,
	TypeHelper.torch_type_to_numpy_type(tensor.dtype),
	[1] if len(tensor.shape) == 0 else list(tensor.shape),
	tensor.data_ptr(),
	)
	# Synchronization are not needed in most cases unless different streams are used or inputs/outputs are in CPU.
	if synchronize:
	self.io_binding.synchronize_inputs()
	self.ort_session.run_with_iobinding(self.io_binding, run_options)
	self.io_binding.synchronize_outputs()
	else:
	self.ort_session.run_with_iobinding(self.io_binding, run_options)
	return self.output_tensors

[hy846130226]

Hi @tianleiwu

Thanks for your help!

I could use the cudaGraphic in tensorrt but I am confused in onnx-tensorrt.

I know I'm supposed to copy the input to the same address, but shouldn't this operation be automated by calling this method?

std::vector Run(const RunOptions& run_options, const char* const* input_names, const Value* input_values, size_t input_count, const char* const* output_names, size_t output_count);

But it seems like I should use some methods to get the IOBinding, then every times I infer the image, I should changed the IOBinding to binding the address, even my address always the same. (every time I got the image data, I will copy it to input address, in other words, my address is reusable)

[hy846130226]

Author

And by the way,
How could I got the IOBinding in onnx-tensorrt in C++.

[tianleiwu]

For cuda graph, you shall only create IO Binding once. For the first call, the cuda graph will be captured. For the remaining calls, you only need copy data to same address and call run with io binding API to replay the captured graph.

An example of I/O binding for TRT in C++ is here:

onnxruntime/onnxruntime/test/shared_lib/test_inference.cc

Lines 1897 to 1909 in 4a196d1

	// Create an OrtValue tensor backed by data on CUDA memory
	Ort::Value bound_y = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(output_data.get()),
	expected_y.size(), expected_y_shape.data(), expected_y_shape.size());
	Ort::IoBinding binding(session);
	binding.BindInput("X", bound_x);
	binding.BindOutput("Y", bound_y);
	// Sychronize to make sure the copy on default stream is done since TensorRT isn't using default stream.
	binding.SynchronizeInputs();
	session.Run(Ort::RunOptions(), binding);
	binding.SynchronizeOutputs();

Example of cuda graph here:

onnxruntime/onnxruntime/test/shared_lib/test_inference.cc

Lines 1975 to 1986 in 4a196d1

	// Enable cuda graph in TRT provider option.
	OrtTensorRTProviderOptionsV2* trt_options;
	ASSERT_TRUE(api.CreateTensorRTProviderOptions(&trt_options) == nullptr);
	std::unique_ptr<OrtTensorRTProviderOptionsV2, decltype(api.ReleaseTensorRTProviderOptions)>
	rel_trt_options(trt_options, api.ReleaseTensorRTProviderOptions);
	std::vector<const char*> keys{"trt_cuda_graph_enable"};
	std::vector<const char*> values{"1"};
	ASSERT_TRUE(api.UpdateTensorRTProviderOptions(rel_trt_options.get(), keys.data(), values.data(), keys.size()) == nullptr);
	ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_TensorRT_V2(
	static_cast<OrtSessionOptions*>(session_options),
	rel_trt_options.get()) == nullptr);

onnxruntime/onnxruntime/test/shared_lib/test_inference.cc

Lines 2052 to 2081 in 4a196d1

	Ort::IoBinding binding(session);
	binding.BindInput("X", bound_x);
	binding.BindOutput("Y", bound_y);
	// One regular run for necessary memory allocation and graph capturing
	session.Run(Ort::RunOptions(), binding);
	// Check the values against the bound raw memory (needs copying from device to host first)
	std::array<float, 3 * 2> y_values;
	(void)cudaMemcpy(y_values.data(), output_data.get(), sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
	ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
	// Replay the captured CUDA graph
	session.Run(Ort::RunOptions(), binding);
	(void)cudaMemcpy(y_values.data(), output_data.get(), sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
	ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
	// Change the input and replay the CUDA graph again.
	x_values = {10.0f, 20.0f, 30.0f, 40.0f, 50.0f, 60.0f};
	(void)cudaMemcpy(input_data.get(), x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);
	binding.SynchronizeInputs();
	session.Run(Ort::RunOptions(), binding);
	(void)cudaMemcpy(y_values.data(), output_data.get(), sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
	expected_y = {10.0f, 40.0f, 90.0f, 160.0f, 250.0f, 360.0f};
	ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
	// Clean up
	binding.ClearBoundInputs();
	binding.ClearBoundOutputs();

[hy846130226]

Hi @tianleiwu

Thanks for your help!

I modify the code according the example but it does not work.

do I miss something?

[tianleiwu]

@hy846130226, please bind inputs and outputs to buffers in GPU memory instead of CPU memory.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.