C++ CUDA Project in Windows - Second call to DNN Module::forward() throws exception

[rtrahms]

🐛 Bug

Environment/Setup

Using libtorch 1.7.0 / CUDA 10.1 / MS Visual Studio 2019

Using torchscript model pt file.

To Reproduce

To reproduce:

1. Confirm CUDA device is present by checking torch::cuda::is_available() and torch::cuda::device_count()

2. Load a model using the torch::jit::script::Module class and torch::jit::load() to specify the pt torchscript file to load.

3. Specify module.to(kCUDA) for CUDA DNN

4. Load an image and convert to a tensor:

   half_ = true // CUDA FP16
   cv::cvtColor(img_input, img_input, cv::COLOR_BGR2RGB); // BGR -> RGB
   img_input.convertTo(img_input, CV_32FC3, 1.0f / 255.0f); // normalization 1/255
   auto tensor_img = torch::from_blob(img_input.data, { 1, img_input.rows, img_input.cols, img_input.channels() }).to(kCUDA);

   tensor_img = tensor_img.permute({ 0, 3, 1, 2 }).contiguous(); // BHWC -> BCHW (Batch, Channel, Height, Width)

   if (half_) {
   tensor_img = tensor_img.to(torch::kHalf);
   }
   std::vectortorch::jit::IValue inputs;
   inputs.emplace_back(tensor_img);

5. Perform inference:
   torch::jit::IValue output = module_.forward(inputs);

The first time this forward() call is made, it works. Any subsequent call will throw an exception. When using the kCPU version of this code, no exception is thrown.

Expected behavior

Once a module is loaded, repeated calls to Module::forward() should return without exception.

cc @peterjc123 @maxluk @nbcsm @guyang3532 @gunandrose4u @smartcat2010 @mszhanyi @gmagogsfm

[mszhanyi]

@rtrahms , thank your feedback. Could you attach your example (.py and .pt) that I could reproduce it easily?

[rtrahms]

Not a python file but a libtorch cpp file, and a supporting Detector class cpp/h files. Built into a MS Visual Studio 2019 project and linking with libtorch libraries.
libTorchTestApp4.zip

[rtrahms]

CUDA PT file for the network attached.
yolov5s_ob12.torchscript-cuda.zip

Labels file as well.
labels_ob_12.txt

[mszhanyi]

CUDA PT file for the network attached.
yolov5s_ob12.torchscript-cuda.zip

Labels file as well.
labels_ob_12.txt

@rtarquini , thank your reply. could you share the image file

[rtrahms]

It is just a standard JPG image loaded with OpenCV. You can use any image file really. The point is it shouldn't crash on feeding the image into forward() on the second pass.

[mszhanyi]

@rtrahms, I tried your example and it passed.
I used py1.7.0+cu110 because there's cu110 in my local machine. opencv is opencv-4.5.0. vs2019 version is 16.7.7.
For convenience, I added torch path and opencv path in PATH.

The test result is

The cmakelist file is attached.
CMakeLists.txt

The image I used is Lenna

[mszhanyi]

@rtrahms , I double checked with https://download.pytorch.org/libtorch/cu101/libtorch-win-shared-with-deps-debug-1.7.0%2Bcu101.zip. It passed too.

[mszhanyi]

Close this issue since it 's not a bug. @rtrahms , feel free to reopen it if you have any new findings

[DBraun]

@rtrahms What was the solution?