Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading opencv image to pytorch tensor #14330

Closed
jainshobhit opened this issue Nov 23, 2018 · 10 comments

Comments

Projects
None yet
6 participants
@jainshobhit
Copy link

commented Nov 23, 2018

Questions and Help

I am trying to load an image in OpenCV Mat variable and then converting it into tensor for passing it into my TorchScript model. I followed #12506 for loading the image however, I am not sure whether it is the correct way or not.
Here is my code

#include <torch/script.h> // One-stop header.
#include <iostream>
#include <memory>
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>

using namespace cv;
using namespace std;

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr << "usage: example-app <path-to-exported-script-module>\n";
    return -1;
  }
  // Deserialize the ScriptModule from a file using torch::jit::load().
  std::shared_ptr<torch::jit::script::Module> module = torch::jit::load(argv[1]);

  assert(module != nullptr);
  std::cout << "ok\n";

  Mat image_bgr, image;
  image_bgr = imread("/home/landscape_org.jpg");
  cvtColor(image_bgr, image, COLOR_BGR2RGB);
  for (int j=0;j<10;j++)
  {
    cout<<image.at<Vec3b>(0,j)<<endl;
  }
  at::Tensor tensor_image = torch::from_blob(image.data, {1, 3, image.rows, image.cols}, at::kByte);
  tensor_image = tensor_image.to(at::kFloat);
  cout<<tensor_image.slice(2,0,1)<<endl;

  // Create a vector of inputs.
  std::vector<torch::jit::IValue> input;
  input.emplace_back(tensor_image);

  // Execute the model and turn its output into a tensor.
  auto output = module->forward(input).toTuple()->elements()[6].toTensor().clone().clamp(0,255);
  Mat output_mat(cv::Size(1920,1080), CV_8UC3, output.data<float>());

  Mat output8, output_bgr;
  cvtColor(output8, output_bgr, COLOR_RGB2BGR);
  imwrite("landscape_output.jpg", output_bgr);
}

The output for the first 10 pixel values is

[53, 149, 249] 
[52, 148, 248] 
[53, 149, 249] 
[55, 151, 251]
[58, 154, 254]
[58, 154, 254]
[61, 155, 255]
[61, 155, 255]
[58, 152, 252]
[58, 152, 252]

And the output on calling the slice function on the resulting tensor (cout<<tensor_image.slice(2,0,1)<<endl;) is (only mentioning the first few columns of the R color channel):

(1,1,.,.) =
Columns 1 to 15   53  149  249   52  148  248   53  149  249   55  151  251   58  154  254
Columns 16 to 30   58  154  254   61  155  255   61  155  255   58  152  252   58  152  252

This clearly shows that the values are not being copied correctly in the tensor.

I am not able to figure out what is the correct way to perform this step since I was not able to find adequate documentation.

@goldsborough

@SsnL

This comment has been minimized.

Copy link
Collaborator

commented Nov 23, 2018

in opencv, the channel dimension is the last dimension. so you would do {1, image.rows, image.cols, 3} in torch::from_blob and transpose afterwards.

@SsnL SsnL closed this Nov 23, 2018

@jainshobhit

This comment has been minimized.

Copy link
Author

commented Nov 23, 2018

Thanks for that information. I had just figured it out.
I am still trying to figure out how to convert the output tensor to OpenCV Mat variable.
Mat output_mat({1920, 1080}, CV_8UC3, output.data<int>()); does not work as mentioned here in Mat constructor documentation.
Haven't been able to find any other documentation yet on this.

@SsnL Any comments?

@bhack

This comment has been minimized.

Copy link

commented Nov 23, 2018

@jainshobhit If you want an 8UC the output is float so you need to convert in at::kByte and access with data<uint8_t>() not with int.

@jainshobhit

This comment has been minimized.

Copy link
Author

commented Nov 26, 2018

Thanks @bhack

@TomHeaven

This comment has been minimized.

Copy link

commented Dec 24, 2018

@SsnL Do you know how to transpose a tensor? I cannot find documentation related to it.

@TomHeaven

This comment has been minimized.

Copy link

commented Dec 24, 2018

I found it at https://pytorch.org/cppdocs/api/classat_1_1_tensor.html#exhale-class-classat-1-1-tensor.
Here is a demo:

 tensor_image = tensor_image.permute({0, 3, 1, 2});
@jainshobhit

This comment has been minimized.

Copy link
Author

commented Dec 27, 2018

A much cleaner way to transpose the axes is also provided in the ATen library. You can find an example for how to use it at https://github.com/jainshobhit/pytorch-cpp-examples/blob/master/libtorch_inference.cpp#L39.
Specifically,
tensor_image = at::transpose(tensor_image, 1, 2);

@martinruenz

This comment has been minimized.

Copy link

commented Feb 20, 2019

@jainshobhit I am curious, why is at::transpose cleaner? Is it faster? I see the advantage of tensor_image.permute that you only need 1 rather than 2 function calls.

@jainshobhit

This comment has been minimized.

Copy link
Author

commented Feb 21, 2019

tensor_image.permute also uses abstractions from the ATen library which makes it slower than at::transpose. Moreover, at::transpose gives you more control over how you wish to swap the axes such that the overall permutation is faster.

@xhuvom

This comment has been minimized.

Copy link

commented Apr 6, 2019

What about the vice-versa? I mean Pytorch tensor array to cv::Mat image?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.