Cannot find DGL C++ sparse library

[mikemhenry]

🐛 Bug

It looks like the logic of dgl/sparse/__init__.py needs to be updated if libdgl_sparse_pytorch_2.0.0.so is ABI compatible with pytorch 2.0.1 or libdgl_sparse_pytorch needs to be rebuilt with pytorch 2.0.1 or the dgl package needs a run_constrained update to indicate it won't work with newer versions of pytorch. These are all the libdgl_sparse_pytorch libs being shipped right now

-rwxrwxr-x 3 mmh mmh 802K May  9 00:48 libdgl_sparse_pytorch_1.12.0.so
-rwxrwxr-x 3 mmh mmh 802K May  9 00:48 libdgl_sparse_pytorch_1.12.1.so
-rwxrwxr-x 3 mmh mmh 843K May  9 00:48 libdgl_sparse_pytorch_1.13.0.so
-rwxrwxr-x 3 mmh mmh 843K May  9 00:48 libdgl_sparse_pytorch_1.13.1.so
-rwxrwxr-x 3 mmh mmh 874K May  9 00:48 libdgl_sparse_pytorch_2.0.0.so

To Reproduce

$ conda install -c dglteam dgl pytorch
$ python
>>> import dgl.sparse as dglsp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mmh/micromamba/envs/dgl-default/lib/python3.11/site-packages/dgl/sparse/__init__.py", line 43, in <module>
    load_dgl_sparse()
  File "/home/mmh/micromamba/envs/dgl-default/lib/python3.11/site-packages/dgl/sparse/__init__.py", line 35, in load_dgl_sparse
    raise FileNotFoundError(f"Cannot find DGL C++ sparse library at {path}")
FileNotFoundError: Cannot find DGL C++ sparse library at /home/mmh/micromamba/envs/dgl-default/lib/python3.11/site-packages/dgl/dgl_sparse/libdgl_sparse_pytorch_2.0.1.so
>>> 

Expected behavior

Environment

• DGL Version (e.g., 1.0): 1.1.0
• Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): pytorch
• OS (e.g., Linux): Linux
• How you installed DGL (conda, pip, source): conda
• Build command you used (if compiling from source):
• Python version: 3.11
• CUDA/cuDNN version (if applicable):
• GPU models and configuration (e.g. V100):
• Any other relevant information:

Additional context

[czkkkkkk]

You can try DGL 1.1.1, which is compatible with Pytorch 2.0.1.

[mikemhenry]

Thanks! Can we keep the issue open until the package is fixed or is this a wont-fix?

[mikemhenry]

Also We are really close to getting dgl on conda-forge which I'll make sure sets this up correctly.

[czkkkkkk]

Thanks! Can we keep the issue open until the package is fixed or is this a wont-fix?

I think we would not fix this issue in a short time. Basically we only support one minor version for each Pytorch major version.

[mikemhenry]

Sounds good, thanks!

[CarlinLiao]

I'm still getting this issue with DGL 1.1.2 and torch 2.1.1

$ python
Python 3.11.5 (main, Sep 11 2023, 13:23:44) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dgl import graph
>>> g = graph(([0, 1, 2], [1, 2, 3]))
>>> g.adj()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/carlin/anaconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/heterograph.py", line 3821, in adj
    from .sparse import spmatrix
  File "/home/carlin/anaconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/sparse/__init__.py", line 43, in <module>
    load_dgl_sparse()
  File "/home/carlin/anaconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/sparse/__init__.py", line 35, in load_dgl_sparse
    raise FileNotFoundError(f"Cannot find DGL C++ sparse library at {path}")
FileNotFoundError: Cannot find DGL C++ sparse library at /home/carlin/anaconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/dgl_sparse/libdgl_sparse_pytorch_2.1.1.so
$ conda list | grep -e 'dgl' -e 'torch'
dgl                       1.1.2.cu118             py311_0    dglteam/label/cu118
ffmpeg                    4.3                  hf484d3e_0    pytorch
libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
pyg                       2.4.0           py311_torch_2.1.0_cu118    pyg
pytorch                   2.1.1           py3.11_cuda11.8_cudnn8.7.0_0    pytorch
pytorch-cuda              11.8                 h8dd9ede_2    pytorch
pytorch-mutex             1.0                        cuda    pytorch
torchtriton               2.1.0                     py311    pytorch
torchvision               0.15.2          cuda118py311h4cc2eb7_0  

• DGL 1.1.2.cu118
• torch 2.1.1
• Python 3.11.5
• conda
• WSL2

[mikemhenry]

@CarlinLiao what is the output of:

$ ls -l /home/carlin/anaconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/dgl_sparse/

[CarlinLiao]

This is from another machine that I've been using with SPT with a slightly different environment (Linux native instead of WSL, Python 3.11.0 instead of 3.11.5), but I was able to replicate the issue. It looks like I have libdgl_spare for torch 2.0.0 and 2.0.1 but not 2.1.1.

$ python
Python 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dgl import graph
>>> g = graph(([0, 1, 2], [1, 2, 3]))
>>> g.adj()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/liaoc2/miniconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/heterograph.py", line 3821, in adj
    from .sparse import spmatrix
  File "/home/liaoc2/miniconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/sparse/__init__.py", line 43, in <module>
    load_dgl_sparse()
  File "/home/liaoc2/miniconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/sparse/__init__.py", line 35, in load_dgl_sparse
    raise FileNotFoundError(f"Cannot find DGL C++ sparse library at {path}")
FileNotFoundError: Cannot find DGL C++ sparse library at /home/liaoc2/miniconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/dgl_sparse/libdgl_sparse_pytorch_2.1.1.so
>>> exit()
$ conda list | grep -e 'dgl' -e 'torch'
dgl                       1.1.2.cu118             py311_0    dglteam/label/cu118
pytorch                   2.1.1           py3.11_cuda11.8_cudnn8.7.0_0    pytorch
pytorch-cuda              11.8                 h7e8668a_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
torchtriton               2.1.0                     py311    pytorch
$ ls -l /home/liaoc2/miniconda3/envs/spt_cggnn/lib/python3.11/site-packages/dgl/dgl_sparse/
total 1752
-rwxrwxr-x 2 liaoc2 liaoc2 894033 Aug 14 04:36 libdgl_sparse_pytorch_2.0.0.so
-rwxrwxr-x 2 liaoc2 liaoc2 894033 Aug 14 04:36 libdgl_sparse_pytorch_2.0.1.so

[czkkkkkk]

Hi @mikemhenry and @CarlinLiao, DGL sparse isn't compatible with Pytorch 2.1.1 for now. Could you try 2.0.x? In fact, for every version update of Pytorch, DGL needs to trigger a corresponding update.