Windows MSVC - scatter.h - unresolved external symbol

[GericoVi]

Hi, I'm trying to load a model with scatter operations within C++. I've successfully built and installed pytorch_scatter using CMake through Visual Studio 2019 and can successfully import it. However when trying to build I get the following error:
Error LNK2001 unresolved external symbol "__int64 __cdecl scatter::cuda_version(void)" (?cuda_version@scatter@@YA_JXZ)

Code:

#include <torch/script.h> // One-stop header.
#include <iostream>
#include <memory>
#include <torch/torch.h>
#include <torchscatter/scatter.h>

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr << "usage: example-app <path-to-exported-script-module>\n";
        return -1;
    }

    std::cout << argv[1] << "\n";


    torch::jit::script::Module module;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load(argv[1]);
    }
    catch (const c10::Error& e) {
        std::cerr << "error loading the model\n";
        std::cout << e.what() << std::endl;
        return -1;
    }

    std::cout << "ok\n";
}

Note that it builds fine without the #include <torchscatter/scatter.h> line. And the resulting executable is able to load a model from torchvision, i.e. the same as this example. But it quietly crashes when I try to load the model.pt that is created by the save_model.py from this example

The CMake method from the PyG examples seems to not work for me - it wouldn't find the scatter.h at all. So I'm using a standard VS project and have the pytorch scatter included within the C/C++ -> Additional Include Directories setting. Also within the Linker -> Additional Library Directories. Followed this for setting up a build environment for torch in VS.

Issue #212 seems related but they seem to switch to the CMake method halfway through before getting it fixed.

Any insight on the unresolved external symbol error? The pytorch scatter was built with the WITH_CUDA flag set to false/off so unsure why it's complaining about it? The resulting directory of the install only has scatter.h and the cpu folder.

[rusty1s]

Interesting. I think for Windows support, there is some additional work to do here (take a look at the torchvision source code for that). We recently started integrating this in torch-sparse (see here), but haven't done that yet for torch-scatter. Pinging @dfalbel for more information.

[GericoVi]

So I'm assuming something like the VISION_API and SPARSE_API definition needs to be made in the header for torch-scatter? And to do that, something like this pull request needs to be done but for torch-scatter?

[rusty1s]

Yes, I think so :)

[dfalbel]

You might also be missing the note:

There is one crucial detail embedded in the above example: The -Wl,--no-as-needed prefix to the warp_perspective link line. This is required because we will not actually be calling any function from the warp_perspective shared library in our application code. We only need the TORCH_LIBRARY function to run. Inconveniently, this confuses the linker and makes it think it can just skip linking against the library altogether. On Linux, the -Wl,--no-as-needed flag forces the link to happen (NB: this flag is specific to Linux!). There are other workarounds for this. The simplest is to define some function in the operator library that you need to call from the main application. This could be as simple as a function void init(); declared in some header, which is then defined as void init() { } in the operator library. Calling this init() function in the main application will give the linker the impression that this is a library worth linking against. Unfortunately, this is outside of our control, and we would rather let you know the reason and the simple workaround for this than handing you some opaque macro to plop in your code.

Found in the docs: https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html#using-the-torchscript-custom-operator-in-c

Since the code doesn't directly use any symbol from the DLL the linker might be completely skipping the linking process for scatter, and thus causing the symbol to be missing.

[GericoVi]

@dfalbel I did actually miss that note thank you. However adding a bit of code to call a scatter function:

    torch::Tensor src = torch::tensor({ 0.5, 0.4, 0.1, 0.6 });
    torch::Tensor index = torch::tensor({ 0, 0, 1, 1 });
    std::cout << src << std::endl;
    std::cout << index << std::endl;
    std::cout << scatter_sum(src, index, 0, torch::nullopt, torch::nullopt) << std::endl;

does not get rid of the error. Now it also complains about scatter_sum being an unresolved symbol when trying to build.

I'm assuming it's still missing the dllimport and dllexport lines that have recently been added to torch_sparse.
Side note: I am able to build with #include <torchsparse/sparse.h>, without needing to directly use a symbol from the torchsparse DLL (or using what I think is the equivalent of --no-as-needed in MSVC)

I've attempted to add the DLL code to torchscatter by doing something similar to the code in torchsparse:

#ifdef _WIN32
#if defined(torchsparse_EXPORTS)
#define SCATTER_API __declspec(dllexport)
#else
#define SCATTER_API __declspec(dllimport)
#endif
#else
#define SPARSE_API
#endif

And for example in the same header (scatter.h) adding that macro to a function:

SCATTER_API torch::Tensor scatter_sum(torch::Tensor src, torch::Tensor index, int64_t dim,
                          torch::optional<torch::Tensor> optional_out,
                          torch::optional<int64_t> dim_size);

But when I add that macro to the function definition in scatter.cpp:

SCATTER_API torch::Tensor scatter_sum(torch::Tensor src, torch::Tensor index, int64_t dim,
                          torch::optional<torch::Tensor> optional_out,
                          torch::optional<int64_t> dim_size) {
  return ScatterSum::apply(src, index, dim, optional_out, dim_size)[0];
}

I get an error:

Error	C2491	'scatter_sum': definition of dllimport function not allowed	

I might be doing this wrong, I'm not familiar with this area, but is there anything obvious that's wrong here?

I've since found a workaround instead of using torch_scatter in C++, so fixing this isn't really urgent for me. But I would be open to continuing helping to get windows support for this library.

[dfalbel]

I think the error is in the line:

#if defined(torchsparse_EXPORTS)

Should probably be torchscatter_EXPORTS. This definition is automatically added by CMake when building shared libraries and is based on the CMake project name AFAICT.

Side note: I am able to build with #include <torchsparse/sparse.h>, without needing to directly use a symbol from the torchsparse DLL (or using what I think is the equivalent of --no-as-needed in MSVC)

Ohh interesting! Then yes, I think adding the export/import definitions will fix the issues.

[GericoVi]

Ah yes I missed that bit thank you. Now that conditional changes and the SCATTER_API expands to __declspec(dllexport) instead of __declspec(dllimport) and the solution builds correctly.

Success! Now my code which loads a serialised model builds correctly on MSVC and I am able to load the model.pt created by this example script.

What I did was I added the SCATTER_API macro to all functions declared in the scatter.h header and then also added them to their definitions within several files (scatter.cpp,segment_coo.cpp,segment_csr.cpp,version.cpp).

Side note: While my code builds with no errors when simply including <torchscatter/scatter.h> and <torchsparse/sparse.h>, and can load a standard PyTorch model (for example from the torchvision example; it quietly stops with no error when trying to load a PyG model (model.pt). This is solved with the direct call workaround:

Found in the docs: https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html#using-the-torchscript-custom-operator-in-c

I know this isn't the PyG repo but I thought I'd mention it within this comment, does this example also need the -Wl,--no-as-needed flag in the CMakeLists.txt (or a direct call within the code) to load the example model correctly @rusty1s? When I try it in my build environment the same quiet crash occurs unless a direct call is added to a torch_scatter and torch_sparse API function.

Thanks for the help both :)

[dfalbel]

Awesome! It's nice that it worked now :)

I think it would be nice if you want to PR your changes. I am planning to work more on PyG in the near future and this would save some time when I get into torch scatter.

[GericoVi]

I've created a draft PR with the changes (and added the torchscatter_EXPORTS macro definition within setup.py similar to what was done in torch_sparse.

However it's failing in the windows pip install with the same error that I had at the start:
scatter.obj : error LNK2001: unresolved external symbol "__int64 __cdecl scatter::cuda_version(void)" (?cuda_version@scatter@@YA_JXZ)
It does install with no errors on ubuntu but then fails during the pytest with a similar error:
undefined symbol: _ZN7scatter12cuda_versionEv

Any thoughts?
I'm not very familiar with the installation procedure with setup.py; probably some compiler or linker arguments missing but unsure which ones or where to insert it. I will continue working on this occasionally over the next few days.