Compilation error when using pybind11 packaged with Libtorch v1.7.1-cuda

[namehta4]

Hi,

I am using pybind11 in my work to send kokkos views from C++ to pytorch. The pybind11 version I am using is a part of the libtorch libraries downloaded from the pytorch website. I am using version 1.7.1-cuda version of libtorch.
I was able to successfully transfer kokkos views to pytorch when using libtorch V1.5.0-cuda with cuda 10.2.89. However, because we have to use updated cuda (< 11.0.3), I downloaded the newer version of libtorch. I am using cmake to build my simple application (which as of now is just a demo code)

I am getting the following error when building my code

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function **"pybind11::detail::collect_arguments" matches the argument list
            argument types are: (const char *const)
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<const char *const &>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(1596): here
            instantiation of "__nv_bool pybind11::detail::object_api<Derived>::contains(T &&) const [with Derived=pybind11::handle, T=const char *const &]" 
/global/u1/n/namehta4/libtorch/include/pybind11/detail/internals.h(271): here**

Could I please request for some assistance regarding this issue?
My code is located at the following githuib link:
[https://github.com/namehta4/KokkosToPytorch]

Thank you!

EDIT (@YannickJadoul): Format code.

[YannickJadoul]

Please follow the issue template and nicely format your message.

Also, what's the full error?

Also also, there's Gitter as well for support questions that aren't really issues with pybind11, as mentioned in the issue template.

[namehta4]

Hi Yannick,

I apologize for not formatting my message according to the issue template.
Should I delete the issue and post it in gitter?
Thank you!

P.S: I was able to isolate the issue. The code works with gcc/7.4.1 but not with gcc/8.3.0
The full error is

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (const char *const)
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<const char *const &>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(1596): here
            instantiation of "__nv_bool pybind11::detail::object_api<Derived>::contains(T &&) const [with Derived=pybind11::handle, T=const char *const &]" 
/global/u1/n/namehta4/libtorch/include/pybind11/detail/internals.h(271): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::handle)
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::handle &>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(976): here
            instantiation of "pybind11::str pybind11::str::format(Args &&...) const [with Args=<pybind11::handle &>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(860): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::handle, pybind11::handle, pybind11::none, pybind11::str)
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::handle, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::handle, pybind11::handle, pybind11::none, pybind11::str>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(1120): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (const char [9])
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<const char (&)[9]>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(1596): here
            instantiation of "__nv_bool pybind11::detail::object_api<Derived>::contains(T &&) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, T=const char (&)[9]]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(1166): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::object, pybind11::str, pybind11::int_)
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::object &, pybind11::str, pybind11::int_>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(976): here
            instantiation of "pybind11::str pybind11::str::format(Args &&...) const [with Args=<pybind11::object &, pybind11::str, pybind11::int_>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(1595): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::cpp_function)
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::handle, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::cpp_function>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(1599): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::object, pybind11::str)
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::object &, pybind11::str>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(976): here
            instantiation of "pybind11::str pybind11::str::format(Args &&...) const [with Args=<pybind11::object &, pybind11::str>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(1604): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::cpp_function, pybind11::none, pybind11::none, const char [1])
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::handle, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::cpp_function, pybind11::none, pybind11::none, const char (&)[1]>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(1608): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::tuple)
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::tuple &>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(2019): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::object)
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::object &>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(2037): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::object)
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::object>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(2038): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pybind11.h(2041): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (pybind11::str)
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::handle, policy=pybind11::return_value_policy::automatic_reference, Args=<pybind11::str>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/numpy.h(467): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (const char [13])
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<const char (&)[13]>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(1596): here
            instantiation of "__nv_bool pybind11::detail::object_api<Derived>::contains(T &&) const [with Derived=pybind11::handle, T=const char (&)[13]]" 
/global/u1/n/namehta4/libtorch/include/pybind11/eval.h(25): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (const char *)
          detected during:
            instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<const char *&>]" 
/global/u1/n/namehta4/libtorch/include/pybind11/pytypes.h(1596): here
            instantiation of "__nv_bool pybind11::detail::object_api<Derived>::contains(T &&) const [with Derived=pybind11::handle, T=const char *&]" 
/global/u1/n/namehta4/libtorch/include/pybind11/embed.h(157): here

/global/u1/n/namehta4/libtorch/include/pybind11/cast.h(2187): error: no instance of overloaded function "pybind11::detail::collect_arguments" matches the argument list
            argument types are: (int, const char [49])
          detected during instantiation of "pybind11::object pybind11::detail::object_api<Derived>::operator()(Args &&...) const [with Derived=pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>, policy=pybind11::return_value_policy::automatic_reference, Args=<int, const char (&)[49]>]" 

[YannickJadoul]

Should I delete the issue and post it in gitter?

No, it's fine; we're here now anyway.

P.S: I was able to isolate the issue. The code works with gcc/7.4.1 but not with gcc/8.3.0

Do you have a minimal example that reproduces this? I still don't see your own code turning up, there. I'm curious which construct in your code causes it...

[namehta4]

Hi Yannick,

Again, I am very grateful for your help.

My reproducer code is as follows

  #include<torch/torch.h>
  #include<pybind11/pybind11.h>
  #include<pybind11/numpy.h>
  #include<pybind11/embed.h>
  #include<Kokkos_Core.hpp>
  #include<Kokkos_Random.hpp>
  #include<iostream>
  #include<random>
  #include<cmath>
  #include<cstdio>
  
  using namespace torch;
  using namespace std;
  namespace py = pybind11;
  
  int main(int argc, char* argv[])
  {
    Kokkos::initialize(argc,argv);
    {
      py::scoped_interpreter guard{};
      py::module sys = py::module::import("sys");
    }
    Kokkos::finalize();
    return 0;
  }

This code is built using the following cmake

**

cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(KokkosToPytorch CXX)

# find dependencies
find_package(Kokkos REQUIRED)
find_package(Torch REQUIRED)
find_package(Python REQUIRED COMPONENTS Interpreter Development)

# targets
file(GLOB SOURCES "src/*.cpp")
add_executable(FirstNN ${SOURCES})

# properties
# C++ properties: at least a C++14 capable compiler is needed
target_compile_features(FirstNN PUBLIC cxx_std_14)
set_target_properties(FirstNN PROPERTIES
    CXX_EXTENSIONS OFF 
    CXX_STANDARD_REQUIRED ON
)

# link/include
target_compile_definitions(FirstNN PRIVATE -DCUSTOM_SYS_PATH="${KokkosToPytorch_SOURCE_DIR}/include")
target_include_directories(FirstNN SYSTEM PRIVATE ${PYTHON_INCLUDE_DIRS})
target_include_directories(FirstNN PRIVATE $<BUILD_INTERFACE:${KokkosToPytorch_SOURCE_DIR}/include>)
target_link_libraries(FirstNN PRIVATE Kokkos::kokkos)
target_link_libraries(FirstNN PRIVATE "${TORCH_LIBRARIES}")
target_link_libraries(FirstNN PRIVATE Python::Python)

**

The cmake command is as follows
CXX=/global/homes/n/namehta4/kokkos/install_cuda/bin/nvcc_wrapper cmake -DCMAKE_PREFIX_PATH="/global/homes/n/namehta4/kokkos/install_cuda;/global/homes/n/namehta4/libtorch" ../

I have following modules listed
cmake/3.14.4, cuda/11.0.3, cudnn/8.0.5,gcc/8.3.0

I use the kokkos cuda built nvcc_wrapper, which is located in my home directory.

Once again, thank you for your help!

[YannickJadoul]

My reproducer code is as follows

And this is failing with the error message from above?

It's still not entirely minimal, as you still depend on this library. Do you need to include these headers to make things fail?

Is it possible to still remove Kokkos and still get a failure?

I use the kokkos cuda built nvcc_wrapper, which is located in my home directory.

Can you make a gist of the full build, including all error messages, then?

[namehta4]

Hi Yannick,

And this is failing with the error message from above?
Yes, the code I have attached generated the error

After following your directions, I removed all the references to kokkos, and there was no error
This is the code

  #include<torch/torch.h>
  #include<pybind11/pybind11.h>
  #include<pybind11/numpy.h>
  #include<pybind11/embed.h>
  
  using namespace torch;
  using namespace std;
  namespace py = pybind11;
  
  int main(int argc, char* argv[])
  {
    py::scoped_interpreter guard{};
    py::module sys = py::module::import("sys");
    return 0;
  }

So the issue is probably how the kokkos nvcc_wrapper interprets the pybind11 directives?
Once again, thank you so much for helping me out.

[YannickJadoul]

I don't know. But that seems like a good guess, yes.

1. In the last example, where things work, did you compile with "the kokkos nvcc_wrapper"? If not, does it fail if the code doesn't mention kokkos, but you compile with kokkos nvcc_wrapper?
2. Can you upload the full output log of the error (for example as a gist), as I asked before? The errors you've shown still miss something, I think?

PS: Multiline code on GitHub uses three times `, so ``` (see https://docs.github.com/en/free-pro-team@latest/github/writing-on-github/basic-writing-and-formatting-syntax#quoting-code)

[namehta4]

Hi Yannick,

1. In the last example(without kokkos directives), the code works even when compiling with nvcc_wrapper. I tried it with and without and it works both ways. So it is not the kokkos nvcc_wrapper causing this issue.
2. Sorry about missing to upload the entire error. As you suggested, I created the gist with the full error, along with the simplest possible code and cmake file used to build the code. The offending cause for this error appears to be the last line in the CMakeLists.txt file, where we specify target link libraries for kokkos. The gist is at ErrorGist

Sorry about not formatting the code attachment properly in my previous message.
Thank you once again!

[YannickJadoul]

@namehta4

1. Ah :-/ So it must be something in those headers :-/ Do you have a bunch of macros or so?
2. Apologies; seems that what you posted was the full error... This is very puzzling! I don't see which part of your code is causing that, immediately :-(

[YannickJadoul]

To be honest, I can't make sense of it.

First of all, collect_arguments is right there and should match. Second, normally GCC gives better errors. So I'm assuming NVCC is messing that up? Is there a way to get better/larger errors? Normally GCC tells you why it can't match each candidate.

Also, which version of pybind11 are you using?

[namehta4]

Hi Yannick,

First of all, thank you so much for going above and beyond to help me with this issue.
I still cant figure out what the problem is. I tried using the --verbose=1 while building the application with cmake and the error message still was the same.

I am using pybond11 which came packaged with the libtorch library available from the pytorch website. I am using libtorch v1.7.1-gpu.

Upon additional analysis, I found out that the error crops up only when using nvcc_wrapper from the kokkos cuda build and not the hsw (haswell) build.
I am providing the build recipes for kokkos so that if anyone wants to replicate the error, they are able to. I am using cuda/11.0.3, cudnn/8.0.5, and gcc/8.3.0.
The recipe I used to build kokkos hsw is as follows:
cmake -DCMAKE_INSTALL_PREFIX=$PWD/../install_hsw -DCMAKE_CXX_COMPILER=g++ -DCMAKE_BUILD_TYPE=Release -DKokkos_ENABLE_OP ENMP=ON -DKokkos_ARCH_HSW=ON -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_TESTS=ON ../
and the recipe I used for building kokkos cuda is as follows:
cmake -DCMAKE_INSTALL_PREFIX=$PWD/../install_cuda -DCMAKE_CXX_COMPILER=$PWD/../bin/nvcc_wrapper -DCMAKE_BUILD_TYPE=Release -DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_VOLTA70=ON -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_CUDA_LAMBDA=ON ../

Once again, sorry for the inconvenience and thank you for you help!

[YannickJadoul]

I am using pybond11 which came packaged with the libtorch library available from the pytorch website.

And which version of pybind11 is this? Can you look that up? (it should be in pybind11/detail/common.h at the top)

Upon additional analysis, I found out that the error crops up only when using nvcc_wrapper from the kokkos cuda build and not the hsw (haswell) build.

Ah, so something's going wrong in that wrapper? Is there a way to turn on/off features one by one?

If not, I'd suggest you go create an issue on Kokkos. If you say everything works without this wrapper, I think we can assume pybind11 is not to blame, here?

[namehta4]

And which version of pybind11 is this? Can you look that up? (it should be in pybind11/detail/common.h at the top)

I am using pybind11 version 2.6.1

Ah, so something's going wrong in that wrapper? Is there a way to turn on/off features one by one?

I am trying to figure out why the two (hsw and cuda kokkos) versions are different.

If not, I'd suggest you go create an issue on Kokkos. If you say everything works without this wrapper, I think we can assume pybind11 is not to blame, here?

I agree. I dont think pybind11 is at fault here and I apologize for raising the issue. I will create an issue with the Kokkos team.
Thank you once again for all your help.

[YannickJadoul]

I am using pybind11 version 2.6.1

Right. So that's the latest one. So I can't tell you to update.

I agree. I dont think pybind11 is at fault here and I apologize for raising the issue. I will create an issue with the Kokkos team.
Thank you once again for all your help.

No worries. Raising it here does make sense, but I don't think there's a lot we can do.

I'll close this, but please reopen if you figure out more and if pybind11 does have an issue to be compiled!