Crash while forwarding model in test_frcnn_tracing from vision/test/tracing/frcnn

[natangold85]

Hi,
I have some problem forwarding the fasterrcnn_resnet50_fpn.pt in test_frcnn_tracing.
I had some problem loading the model, i solved it by adding those lines:

#include <torchvision/empty_tensor_op.h>

static auto registry =
        torch::RegisterOperators().op("torchvision::roi_align(Tensor input, Tensor rois, float spatial_scale, int "
                                      "pooled_height, int pooled_width, int sampling_ratio, bool aligned) -> Tensor",
                                      &roi_align)
                                      .op("torchvision::nms", &nms)
                                      .op("torchvision::_new_empty_tensor_op", &new_empty_tensor);

after solving this problem i'm encountering a segmentation fault in the forward function.
its looking like the segmentation fault is coming from torchvision/nms.h in this call:

return op.call(dets, scores, iou_threshold);

my torchvision is compiled from source in branch v0.8.2
my libtorch is also compiled from source in branch v1.7.1

To compile the frcnn model i'm using python 3.6.8
with pytorch 1.7.1 and torchvision 0.8.2

Can you please help me understand what is the problem here?
Thanks.

[bmanga]

Hi @natangold85 can you try with the latest torchvision master? As I'm guessing you're on windows, did you use the same configuration for all libraries (debug/release) as your final executable?

[natangold85]

No, Im on Linux ubuntu 16.
Both torchvision and libtorch was debug.
I will try with latest and report.

Regarding the symbols loading I needed to do to pass model loading, that get me a little suspicious that something is different in the compilation.
I saw this dynamic loading on different github issue. Do you know why I and others needed to do it but in the frcnn exmple its not done?

[fmassa]

@natangold85 please let us know if you still face the same issue after updating to use latest master from PyTorch and torchvision

[natangold85]

Will update. I have some non-related tasks and will get back to it.

[natangold85]

Hi updating that taking nightly torch and nightly torchvision seems to change something but i'm not sure if it improved.
this is an issue i added to pytorch about their nightly here: pytorch/pytorch#52674
trying to take torchvision nightly and libtorch 1.7.1 failed on error attached below.
So i can't take working libtorch and nightly torchvision.

You have any suggestion regarding what can I do here?

Thanks

../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::detail::infer_schema::make_function_schema(c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::RecordFunction::RecordFunction(at::RecordScope, bool)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::empty(c10::ArrayRef<long>, c10::TensorOptions, c10::optional<c10::MemoryFormat>)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, char const*)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::zeros(c10::ArrayRef<long>, c10::TensorOptions)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::Tensor::__dispatch_contiguous(c10::MemoryFormat) const'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::impl::OperatorEntry::reportSignatureError(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::shouldRunRecordFunction(bool*)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::Dispatcher::realSingleton()'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::from_file(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10::optional<bool>, c10::optional<long>, c10::TensorOptions)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `caffe2::TypeMeta::error_unsupported_typemeta(caffe2::TypeMeta)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::impl::raw_local_dispatch_key_set'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::Dispatcher::runRecordFunction(at::RecordFunction&, c10::OperatorHandle const&, c10::DispatchKey, std::vector<c10::IValue, std::allocator<c10::IValue> >&&)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::ones_like(at::Tensor const&, c10::TensorOptions, c10::optional<c10::MemoryFormat>)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `torch::autograd::ForwardADLevel::try_get_by_idx(unsigned long)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `at::zeros_like(at::Tensor const&, c10::TensorOptions, c10::optional<c10::MemoryFormat>)'
../../../../../../embedded_algo/thirdParty/torchvision/lib/libtorchvision.so: undefined reference to `c10::Dispatcher::runRecordFunction(at::RecordFunction&, c10::OperatorHandle const&, c10::DispatchKey)'

[bmanga]

Could it be that you used a pre-built libtorch with pre-C++11 ABI?

[natangold85]

it just a slightly older version built in the same way.
simple cmake command.
I took the v1.7.1 branch and built it.
But I will verify all my steps again

[natangold85]

I used master libtorch to compile torchvision.
so i observed this error when linking torchvision against the version that he wasn't compile with.
when i use libtorch from v1.7.1 to compile updted torchvision i get this error:

/data2/torchvision/vision/torchvision/csrc/ops/deform_conv2d.cpp:86:23: error: expected constructor, destructor, or type conversion before ‘(’ token
TORCH_LIBRARY_FRAGMENT(torchvision, m) {
^
CMakeFiles/torchvision.dir/build.make:458: recipe for target 'CMakeFiles/torchvision.dir/torchvision/csrc/ops/deform_conv2d.cpp.o' failed
make[2]: *** [CMakeFiles/torchvision.dir/torchvision/csrc/ops/deform_conv2d.cpp.o] Error 1
CMakeFiles/Makefile2:94: recipe for target 'CMakeFiles/torchvision.dir/all' failed
make[1]: *** [CMakeFiles/torchvision.dir/all] Error 2
Makefile:148: recipe for target 'all' failed
make: *** [all] Error 2

To conclude, updated torchvision with updated libtorch not working due to jit error (pytorch/pytorch#52674 ).
0.8.2 torchvision with 1.7.1 libtorch not working because of the error in this issue.
And i can't compile updated torchvision with working libtorch.

have any suggestion what can i do to solve it?

[natangold85]

Wanted to update that i found the problem in libtorch. So i'm using current master for libtorch and torchvision and i'm able to use torchvision operators on jit.

Thanks !