Improve detection/packaging with system-wide Magma

[mratsim]

Hello team,

I have trouble making PyTorch works properly with a systemwide Magma install on Archlinux.
Magma is installed at the following location:

$  pacman -Ql magma|grep 'libmagma.so\|magma.h'
magma /opt/magma/include/magma.h
magma /opt/magma/lib/libmagma.so

This is the build script I'm using: https://github.com/mratsim/Arch-Data-Science/blob/master/python-pytorch-git/PKGBUILD
Especially this part to add my directory to Magma detection routine:

  #Adding MAGMA on Archlinux support
  sed -i 's#/usr/local/magma/lib#/opt/magma/lib#g' "${_pkgname}/torch/lib/THC/cmake/FindMAGMA.cmake"
  sed -i 's#/usr/local/magma/include#/opt/magma/include#g' "${_pkgname}/torch/lib/THC/cmake/FindMAGMA.cmake"
  sed -i 's#/usr/local/magma/lib#/opt/magma/lib#g' "${_pkgname}/torch/lib/THCS/cmake/FindMAGMA.cmake"
  sed -i 's#/usr/local/magma/include#/opt/magma/include#g' "${_pkgname}/torch/lib/THCS/cmake/FindMAGMA.cmake"

During compilation magma.h is properly found however on importing torch I get the following error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-cf1060cf0cd9> in <module>()
----> 1 import torch
      2 from torch.autograd import Variable

/usr/lib/python3.6/site-packages/torch/__init__.py in <module>()
     51 sys.setdlopenflags(_dl_flags.RTLD_GLOBAL | _dl_flags.RTLD_NOW)
     52 
---> 53 from torch._C import *
     54 
     55 __all__ += [name for name in dir(_C)

ImportError: libmagma.so: cannot open shared object file: No such file or directory

And indeed a ldd on _C gives the following:

$  ldd _C.cpython-36m-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fffedf67000)
        libshm.so => /usr/lib/python3.6/site-packages/torch/./lib/libshm.so (0x00007fb762f19000)
        libcudart.so.8.0 => /opt/cuda/lib64/libcudart.so.8.0 (0x00007fb762cb3000)
        libcudnn.so.5 => /opt/cuda/lib64/libcudnn.so.5 (0x00007fb75da44000)
        libpython3.6m.so.1.0 => /usr/lib/libpython3.6m.so.1.0 (0x00007fb75d4c7000)
        libTH.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libTH.so.1 (0x00007fb75ce7a000)
        libTHS.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libTHS.so.1 (0x00007fb75cc4d000)
        libTHPP.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libTHPP.so.1 (0x00007fb75c93f000)
        libTHNN.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libTHNN.so.1 (0x00007fb75c63f000)
        libTHC.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libTHC.so.1 (0x00007fb751cd9000)
        libTHCS.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libTHCS.so.1 (0x00007fb751ac3000)
        libTHCUNN.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libTHCUNN.so.1 (0x00007fb74f31b000)
        libnccl.so.1 => /usr/lib/python3.6/site-packages/torch/./lib/libnccl.so.1 (0x00007fb74d44e000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fb74d235000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fb74d017000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fb74cc73000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x0000560721b76000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007fb74ca6b000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fb74c6e3000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007fb74c3d0000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fb74c1ca000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007fb74bfc7000)
        libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007fb74afa6000)
        libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007fb74ad78000)
        libcublas.so.8.0 => /opt/cuda/lib64/libcublas.so.8.0 (0x00007fb7482e0000)
        libcurand.so.8.0 => /opt/cuda/lib64/libcurand.so.8.0 (0x00007fb744368000)
        libmagma.so => not found
        libcusparse.so.8.0 => /opt/cuda/lib64/libcusparse.so.8.0 (0x00007fb74184d000)
        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00007fb74151c000)
        libquadmath.so.0 => /usr/lib/../lib/libquadmath.so.0 (0x00007fb7412da000)

Would it be possible to add an environment variable for Magma like there are for Cuda and CuDNN (CUDNN_LIB_DIR and CUDNN_INCLUDE_DIR)

Thanks

[albanD]

Isn't LD_LIBRARY_PATH the variable your looking for?

[mratsim]

I've fixed the linking issue on my machine by adding the Magma libs path to /etc/ld.so.conf.d/, thanks for the hint.

I do think there is a need of a MAGMA_LIB_DIR and MAGMA_INCLUDE_DIR so that people who don't install magma in /usr/local don't have to sed the FindMAGMA files like the following

#Adding MAGMA on Archlinux support
  sed -i 's#/usr/local/magma/lib#/opt/magma/lib#g' "${_pkgname}/torch/lib/THC/cmake/FindMAGMA.cmake"
  sed -i 's#/usr/local/magma/include#/opt/magma/include#g' "${_pkgname}/torch/lib/THC/cmake/FindMAGMA.cmake"
  sed -i 's#/usr/local/magma/lib#/opt/magma/lib#g' "${_pkgname}/torch/lib/THCS/cmake/FindMAGMA.cmake"
  sed -i 's#/usr/local/magma/include#/opt/magma/include#g' "${_pkgname}/torch/lib/THCS/cmake/FindMAGMA.cmake"

[apaszke]

Isn't it enough to add magma dirs to CMAKE_PREFIX_PATH at build time?

[mratsim]

I tried by using export CMAKE_PREFIX_PATH="/opt/magma/include;/opt/magma/lib" before building or changing the build script to:

   CC=gcc-5 \
   CXX=g++-5 \
   WITH_CUDA=1 \
   CUDA_HOME=/opt/cuda \
   WITH_CUDNN=1 \
   CUDNN_LIB_DIR=/opt/cuda/lib64 \
   CUDNN_INCLUDE_DIR=/opt/cuda/include \
   CMAKE_PREFIX_PATH="/opt/magma/include;/opt/magma/lib" \
   python setup.py build

Only sed-ing the FindMAGMA file works for me

[ngimel]

Have you tried CMAKE_PREFIX_PATH=/opt/magma, because that's how it should be set? lib and include suffixes are added by FIND_LIBRARY and FIND_PATH. In Dockerfile CMAKE_PREFIX_PATH is used to locate magma in /opt/conda/envs/pytorch-py35/ without sed'ing FindMAGMA, and it works.

[mratsim]

@ngimel Thank you, it's working 👍 .

So everything is there. Closing this.