Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing an opencv GpuMat #13898

Closed
leconteur opened this issue Nov 13, 2018 · 12 comments

Comments

Projects
None yet
5 participants
@leconteur
Copy link

commented Nov 13, 2018

馃悰 Bug

Contrary to opencv Mat objects, GpuMat are strided in memory. It seems impossible to tell the torch::from_blob method from the c++ api this information. We used to build our tensor directly with A10 which provided this options.

To Reproduce

Steps to reproduce the behavior:

#include <torch/script.h> // One-stop header.

#include <iostream>
#include <memory>
#include <opencv2/opencv.hpp>
#include <opencv2/core/cuda.hpp>
//#include "opencv2/gpu/gpu.hpp"

using namespace cv;

int main(int argc, const char* argv[]) {
    Mat gray_image = Mat::ones(9, 9, CV_32FC1); 
    Mat image;
    cvtColor(gray_image, image, COLOR_GRAY2RGB);

    cuda::GpuMat gImage;
    gImage.upload(image);
    std::vector<int64_t> sizes = {static_cast<int64_t>(gImage.channels()),
                              static_cast<int64_t>(gImage.rows),
                              static_cast<int64_t>(gImage.cols)};

    auto options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA);
    auto tensor_image = torch::from_blob(gImage.data, torch::IntList(sizes), options);
    std::cout << tensor_image << std::endl;
    return 0;
}

This produced the following output.

(1,.,.) = 
  1  1  1  1  1  1  1  1  1
  1  1  1  1  1  1  1  1  1
  1  1  1  1  1  1  1  1  1
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0

(2,.,.) = 
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  1  1  1  1  1  1  1
  1  1  1  1  1  1  1  1  1
  1  1  1  1  1  1  1  1  1
  1  1  0  0  0  0  0  0  0

(3,.,.) = 
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
  0  0  0  0  0  0  0  0  0
[ Variable[CUDAFloatType]{3,9,9} ]
Segmentation fault (core dumped)

Expected behavior

If I use only opencv Mat, I get a tensor filled with ones.

Environment

  • PyTorch Version (e.g., 1.0): 1.0 nightly
  • OS (e.g., Linux): Linux
  • How you installed PyTorch (conda, pip, source): pip
  • Build command you used (if compiling from source):
  • Python version: 3.6.5
  • CUDA/cuDNN version: CUDA runtime version: 8.0.61
  • GPU models and configuration: Tesla K40
@soumith

This comment has been minimized.

Copy link
Member

commented Nov 13, 2018

cc: @goldsborough
I guess we need a way to expose strides to from_blob

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Nov 13, 2018

Wait what about this guy:

inline Tensor from_blob(
    void* data,
    IntList sizes,
    IntList strides,
    const std::function<void(void*)>& deleter,
const TensorOptions& options = {})
@leconteur

This comment has been minimized.

Copy link
Author

commented Nov 14, 2018

That seems to be what I am looking for. Is there documentation somewhere on what is the deleter argument?

@leconteur

This comment has been minimized.

Copy link
Author

commented Nov 14, 2018

Thank you for your help. It does work with those modifications. However, I did not manage to make a torch tensor but only an A10 tensor. As such, I can't use it in a model.

#include <torch/script.h> // One-stop header.

#include <iostream>
#include <memory>
#include <opencv2/opencv.hpp>
#include <opencv2/core/cuda.hpp>
#include <ATen/ATen.h>

using namespace cv;

void deleter(void* arg) {};

int main(int argc, const char* argv[]) {
    Mat gray_image = Mat::ones(9, 9, CV_32FC1); 
    Mat image;
    cvtColor(gray_image, image, COLOR_GRAY2RGB);

    cuda::GpuMat gImage;
    gImage.upload(image);
    std::vector<int64_t> sizes = {1, static_cast<int64_t>(gImage.channels()),
                              static_cast<int64_t>(gImage.rows),
                              static_cast<int64_t>(gImage.cols)};
    long long step = gImage.step / sizeof(float);
    std::vector<int64_t> strides = {1, step, static_cast<int64_t>(gImage.channels())};

    auto options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA);
    auto tensor_image = at::from_blob(gImage.data, torch::IntList(sizes), torch::IntList(strides), deleter,  options);
    std::cout << tensor_image << std::endl;
    return 0;
}

@bhack

This comment has been minimized.

Copy link

commented Nov 14, 2018

@leconteur I've added also this documentation request for the Torchscript tutorial at pytorch/tutorials#352. Cause we still have missing docs for regular cv::Mat to be recovered from slides or support issues. I think that this knowledge will be lost if will go on closed issue.

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Nov 14, 2018

@leconteur You need to change at::from_blob to torch::from_blob. That will create a variable instead of a tensor. If you upgrade to the latest nightly you can just use torch:: instead of at:: for everything and avoid such details. So just change

auto options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA);
auto tensor_image = at::from_blob(gImage.data, torch::IntList(sizes), torch::IntList(strides), deleter,  options);

to

auto tensor_image = torch::from_blob(gImage.data, sizes, strides, deleter,  torch::kCUDA);

Notice I removed the explicit at::IntList() conversions -- those happen implicitly. An you can reduce the whole line about creating tensor options to just torch::kCUDA -- float32 is the default already. Also see https://pytorch.org/cppdocs/notes/tensor_creation.html for a tutorial on how to create tensors. You also shouldn't need to include #include <ATen/ATen.h>, that will always be available through the torch headers.

I'll also send a PR to make the deleter an optional argument for the overload that takes strides.

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Nov 14, 2018

Oh wait, I see there is no torch:: version of this overload you need. Let me add that.

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Nov 14, 2018

Fixing in #13982

@jxmelody

This comment has been minimized.

Copy link

commented Nov 20, 2018

hi, @leconteur
I am using OpenCV to load images too, but getting some troubles in cmake & make , my CMakeList.txt looks like this

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(custom_ops)
find_package(OpenCV )
find_package(Torch  )
add_executable(example-app test.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}" "${OPENCV_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 11)
                                                                        

I used cmake -DCMAKE_PREFIX_PATH="/usr/local/cv;../libtorch .." to produce MakeFile.I've got no errors after this, but when I did make, It showed me some strange errors:
fatal error: opencv2/core.hpp: no such a file or directory

Could you please letting me know how you wrote CMakeFile and cmake commands? Or more generally, do you know what is wrong in my situation?

Also I've tired to use gcc or qmake with including the include and libdir ,but it keep turning out this

/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.2/ld: warning: libnvToolsExt-3965bdd0.so.1, needed by /home/xiaojin/workspace/download/libtorch/lib/libtorch.so, not found (try using -rpath or -rpath-link)
/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.2/ld: warning: libcaffe2_gpu.so, needed by /home/xiaojin/workspace/download/libtorch/lib/libtorch.so, not found (try using -rpath or -rpath-link)
/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.2/ld: warning: libc10.so, needed by /home/xiaojin/workspace/download/libtorch/lib/libtorch.so, not found (try using -rpath or -rpath-link)
/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.2/ld: warning: libgomp-7bcb08ae.so.1, needed by /home/xiaojin/workspace/download/libtorch/lib/libcaffe2.so, not found (try using -rpath or -rpath-link)
test.o锛歩n function 鈥榗10::intrusive_ptr_target::~intrusive_ptr_target()鈥欙細
test.cpp:(.text._ZN3c1020intrusive_ptr_targetD2Ev[_ZN3c1020intrusive_ptr_targetD5Ev]+0x126)锛氣榗10::Error::Error(c10::SourceLocation, std::string const&)鈥檜ndefined reference
test.cpp:(.text._ZN3c1020intrusive_ptr_targetD2Ev[_ZN3c1020intrusive_ptr_targetD5Ev]+0x256)锛氣榗10::Error::Error(c10::SourceLocation, std::string const&)鈥檜ndefined reference
test.o锛欼n function 鈥榗10::Device::Device(c10::DeviceType, short)鈥欙細
test.cpp:(.text._ZN3c106DeviceC2ENS_10DeviceTypeEs[_ZN3c106DeviceC5ENS_10DeviceTypeEs]+0xe4)锛氣榗10::Error::Error(c10::SourceLocation, std::string const&)鈥檜ndefined reference
test.cpp:(.text._ZN3c106DeviceC2ENS_10DeviceTypeEs[_ZN3c106DeviceC5ENS_10DeviceTypeEs]+0x1f7)锛氣榗10::Error::Error(c10::SourceLocation, std::string const&)鈥檜ndefined reference

Thanks!

@leconteur

This comment has been minimized.

Copy link
Author

commented Nov 22, 2018

My cmake looked like this.

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(custom_ops)
find_package(Torch REQUIRED)
find_package( OpenCV REQUIRED )

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}" ${OpenCV_LIBS})
set_property(TARGET example-app PROPERTY CXX_STANDARD 11)
@jxmelody

This comment has been minimized.

Copy link

commented Nov 23, 2018

Thanks, @leconteur
I use qmake to compile and succeed .
But I still have one question: Have you load images by OpenCV correctly? I got a tensor filled with ones ( just like you post above).

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Nov 26, 2018

#13982 landed. from_blob with strides is now available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.