Compatible versions of Eigen and Dynet for Cuda 11.1

[vahini01]

Hi,
I couldn't find any compatible versions.

System Specifications:
Cuda Version - 11.1

Tried:

Build Command:
To avoid Unsupported GPU architecture for compute_30 during build time, the below command is used.

cmake .. -DEIGEN3_INCLUDE_DIR=../eigen -DPYTHON='which python' -DBACKEND=cuda -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.1

Dynet	Eigen	Error
Latest(master branch)	Eigen 3.2	CX11 folder is not present
Latest(master branch)	Eigen 3.3	identifier std::round is undefined in device code
Latest(master branch)	Eigen 3.3.7	identifier std::round is undefined in device code
Latest(master branch)	Eigen 3.4	Error while running make
Dynet 2.0.3	Eigen-2355b22	Unsupported GPU architecture for compute_30 (while running make)
Dynet 2.1	Eigen-b2e267dc99d4.zip	Unsupported GPU architecture for compute_30 (while running make)

Does anyone know how to fix the above issue?

[albertbn]

getting same error (in google colab):

/content/dynet-base/dynet/build
[  1%] Building NVCC (Device) object dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge3.cu.o
[  2%] Building NVCC (Device) object dynet/CMakeFiles/dynet.dir/dynet_generated_cudnn-ops.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_30'
nvcc fatal   : Unsupported gpu architecture 'compute_30'
CMake Error at dynet_generated_cudnn-ops.cu.o.Release.cmake:216 (message):
  Error generating
  /content/dynet-base/dynet/build/dynet/CMakeFiles/dynet.dir//./dynet_generated_cudnn-ops.cu.o

CMake Error at dynet_generated_gpu-merge3.cu.o.Release.cmake:216 (message):
  Error generating
  /content/dynet-base/dynet/build/dynet/CMakeFiles/dynet.dir//./dynet_generated_gpu-merge3.cu.o



dynet/CMakeFiles/dynet.dir/build.make:110: recipe for target 'dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge3.cu.o' failed
make[2]: *** [dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge3.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
dynet/CMakeFiles/dynet.dir/build.make:75: recipe for target 'dynet/CMakeFiles/dynet.dir/dynet_generated_cudnn-ops.cu.o' failed
make[2]: *** [dynet/CMakeFiles/dynet.dir/dynet_generated_cudnn-ops.cu.o] Error 1
CMakeFiles/Makefile2:116: recipe for target 'dynet/CMakeFiles/dynet.dir/all' failed
make[1]: *** [dynet/CMakeFiles/dynet.dir/all] Error 2
Makefile:135: recipe for target 'all' failed
make: *** [all] Error 2

---

the below lines are working:

%cd /content
!mkdir dynet-base
%cd dynet-base
!pwd
!git clone https://github.com/clab/dynet.git
!wget https://gitlab.com/libeigen/eigen/-/archive/3.3.4/eigen-3.3.4.tar.bz2
!tar -xf eigen-3.3.4.tar.bz2
!mv eigen-3.3.4 eigen
%cd dynet
!mkdir build
%cd build
!cmake .. -DEIGEN3_INCLUDE_DIR=../../eigen -DPYTHON=`which python` -DBACKEND=cuda

and here it fails:

!make -j 2

[albertbn]

I've managed to setup a working DyNet with GPU version on Google Colab

the colab notebook below has all the installation and some test tutorial snippets from the dynet docs.
https://colab.research.google.com/drive/1HMIhb8-5c-Ds1hF9jklHk_3YzLXTPQJ5?usp=sharing

I've double-confirmed GPU is used by opening a colab terminal and running this command:
watch nvidia-smi

main parts that helped in the source installation script are:

using precisely this Eigen version:
https://github.com/clab/dynet/releases/download/2.1/eigen-b2e267dc99d4.zip

and remove the compute_30 architecture with sed:
sed -i 's/-gencode;arch=compute_30,code=sm_30;//g' dynet/dynet/CMakeLists.txt

make -j 2 - takes around 12 minutes to compile - which is an issue on colab, since it'll happen each time one loads a runtime

python ../../setup.py build --build-dir=.. --skip-build install - takes around 47 seconds

for having dynet globbaly in python - this is what worked for me:

import sys
sys.path.insert(0, '/content/dynet-base/dynet/build/python/')

several resources helped:
this issue and others, dynet repo readme, tutorials - all I've used was resolved/written somewhere - I've just put parts together

[vahini01]

Thanks albertbn! The above versions and steps worked for me.

[g-karthik]

@albertbn @vahini01 I followed the updated steps you shared for manual GPU-installation and that resolved the installation issue for me as well, thank you!

However, I see the following error when trying to run a training job with GPU enabled (--dynet-gpu enabled and --dynet-mem as 3000):

[dynet] random seed: 2959804024
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
[dynet] initializing CUDA
[dynet] CUDA driver/runtime versions are 11.1/11.0
Request for 1 GPU ...
[dynet] Device Number: 0
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device Number: 1
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device Number: 2
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device Number: 3
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device Number: 4
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device Number: 5
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device Number: 6
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device Number: 7
[dynet]   Device name: Tesla V100-SXM2-16GB
[dynet]   Memory Clock Rate (KHz): 877000
[dynet]   Memory Bus Width (bits): 4096
[dynet]   Peak Memory Bandwidth (GB/s): 898.048
CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes )
the provided PTX was compiled with an unsupported toolchain.
[dynet]   FAILED to get free memory
[dynet] Device(s) selected: 0CUDA failure in cudaMalloc(&ptr, n)
the provided PTX was compiled with an unsupported toolchain.
terminate called after throwing an instance of 'dynet::cuda_exception'
  what():  cudaMalloc(&ptr, n)

It's not clear to me why I am facing this error. Do you have any ideas? Have you faced this after installation?

[albertbn]

For these memory issues you should contact the dynet team. I think there might be an open issue on this already. I also experienced a memory/ segmentation fault with a somewhat larger example than the toy 101 example - it fails on CPU as well. I had to switch to another framework to complete my project.

…

On Wed, 26 Oct 2022 at 6:16, Karthik Gopalakrishnan < ***@***.***> wrote: @albertbn <https://github.com/albertbn> @vahini01 <https://github.com/vahini01> I followed the updated steps you shared for manual GPU-installation and that resolved the installation issue for me as well, thank you! However, I see the following error when trying to run a training job with GPU enabled (--dynet-gpu and --dynet-mem as 3000): [dynet] random seed: 2959804024 [dynet] allocating memory: 512MB [dynet] memory allocation done. [dynet] initializing CUDA [dynet] CUDA driver/runtime versions are 11.1/11.0 Request for 1 GPU ... [dynet] Device Number: 0 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device Number: 1 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device Number: 2 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device Number: 3 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device Number: 4 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device Number: 5 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device Number: 6 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device Number: 7 [dynet] Device name: Tesla V100-SXM2-16GB [dynet] Memory Clock Rate (KHz): 877000 [dynet] Memory Bus Width (bits): 4096 [dynet] Peak Memory Bandwidth (GB/s): 898.048 CUDA failure in cudaMemGetInfo( &free_bytes, &total_bytes ) the provided PTX was compiled with an unsupported toolchain. [dynet] FAILED to get free memory [dynet] Device(s) selected: 0CUDA failure in cudaMalloc(&ptr, n) the provided PTX was compiled with an unsupported toolchain. terminate called after throwing an instance of 'dynet::cuda_exception' what(): cudaMalloc(&ptr, n) It's not clear to me why I am facing this error. Do you have any ideas? Have you faced this after installation? — Reply to this email directly, view it on GitHub <#1652 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADJB45YIHFZEIXPGZYXRCD3WFCPABANCNFSM5VT7MLXQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[g-karthik]

@neubig could you please help with this? Running on CPU is taking WAY too long and I would really like to use my GPUs :)