New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't compile with GPU=1 #230

Open
AnanasPie opened this Issue Oct 4, 2017 · 12 comments

Comments

Projects
None yet
9 participants
@AnanasPie

AnanasPie commented Oct 4, 2017

Hi, all,

I am trying to compile YOLO with GPU=1.
I get the following error:
image

I already have done:

  • Install the updated CUDA version (9.0)
  • Add this line inside the makefile - NVCC += -D_FORCE_INLINES

Thanks,
Mark

@Dahlasam

This comment has been minimized.

Show comment
Hide comment
@Dahlasam

Dahlasam Oct 7, 2017

While CPU make succeeds and darknet works well, the GPU=1 make to enable GPU support fails in same way as for @AnanasPie . Other components like tensorflow-gpu are able to use CUDA in the system.

I get similar error with my GPU:

ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 115; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 144; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 227; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 257; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 341; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 361; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 379; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 399; error : Call has wrong number of parameters ptxas fatal : Ptx assembly aborted due to errors Makefile:91: recipe for target 'obj/convolutional_kernels.o' failed make: *** [obj/convolutional_kernels.o] Error 255

System details:
OS: Ubuntu 16.04 x64
GPU: Nvidia GTX 1060
CUDA: 8.0
cuDNN: 6.0
nvidia-smi output:

`+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.81 Driver Version: 384.81 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 106... Off | 00000000:01:00.0 On | N/A |
| 0% 43C P8 7W / 156W | 5955MiB / 6069MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1042 G /usr/lib/xorg/Xorg 197MiB |
| 0 1839 G compiz 77MiB |
| 0 2923 G ...-token=5D46E4CA8207FFF70E348E0EA2F3C753 108MiB |
| 0 28788 C /home/sami/miniconda2/bin/python 5567MiB |
+-----------------------------------------------------------------------------+`

Any suggestions? What CUDA and CUDNN versions are supported?

Thanks,
Sami

Dahlasam commented Oct 7, 2017

While CPU make succeeds and darknet works well, the GPU=1 make to enable GPU support fails in same way as for @AnanasPie . Other components like tensorflow-gpu are able to use CUDA in the system.

I get similar error with my GPU:

ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 115; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 144; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 227; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 257; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 341; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 361; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 379; error : Call has wrong number of parameters ptxas /tmp/tmpxft_00001c51_00000000-15_convolutional_kernels.compute_30.ptx, line 399; error : Call has wrong number of parameters ptxas fatal : Ptx assembly aborted due to errors Makefile:91: recipe for target 'obj/convolutional_kernels.o' failed make: *** [obj/convolutional_kernels.o] Error 255

System details:
OS: Ubuntu 16.04 x64
GPU: Nvidia GTX 1060
CUDA: 8.0
cuDNN: 6.0
nvidia-smi output:

`+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.81 Driver Version: 384.81 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 106... Off | 00000000:01:00.0 On | N/A |
| 0% 43C P8 7W / 156W | 5955MiB / 6069MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1042 G /usr/lib/xorg/Xorg 197MiB |
| 0 1839 G compiz 77MiB |
| 0 2923 G ...-token=5D46E4CA8207FFF70E348E0EA2F3C753 108MiB |
| 0 28788 C /home/sami/miniconda2/bin/python 5567MiB |
+-----------------------------------------------------------------------------+`

Any suggestions? What CUDA and CUDNN versions are supported?

Thanks,
Sami

@Dahlasam

This comment has been minimized.

Show comment
Hide comment
@Dahlasam

Dahlasam Oct 10, 2017

After downgrading to CUDA 8.0 and cuDNN 6.0 I found this hint from Stack Overflow:
Add the following to your ~/.bashrc

# DARKNET
export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

After this change the compiling worked and darknet jumped into CUDASpeed as expected!

Dahlasam commented Oct 10, 2017

After downgrading to CUDA 8.0 and cuDNN 6.0 I found this hint from Stack Overflow:
Add the following to your ~/.bashrc

# DARKNET
export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

After this change the compiling worked and darknet jumped into CUDASpeed as expected!

@j5207

This comment has been minimized.

Show comment
Hide comment
@j5207

j5207 Nov 12, 2017

Hi:
I have the same problem and even after adding these code to ~/.bashrc, it still could not be compiled. I'm using CUDA 7.5
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90 Driver Version: 384.90 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro K620 Off | 00000000:01:00.0 On | N/A |
| 34% 33C P8 1W / 30W | 337MiB / 1992MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1188 G /usr/lib/xorg/Xorg 189MiB |
| 0 1925 G compiz 70MiB |
| 0 2754 G ...-token=02DD3E6D0D22113602503DA6AAD1DD92 74MiB |
+-----------------------------------------------------------------------------+

Thanks!

j5207 commented Nov 12, 2017

Hi:
I have the same problem and even after adding these code to ~/.bashrc, it still could not be compiled. I'm using CUDA 7.5
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90 Driver Version: 384.90 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro K620 Off | 00000000:01:00.0 On | N/A |
| 34% 33C P8 1W / 30W | 337MiB / 1992MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1188 G /usr/lib/xorg/Xorg 189MiB |
| 0 1925 G compiz 70MiB |
| 0 2754 G ...-token=02DD3E6D0D22113602503DA6AAD1DD92 74MiB |
+-----------------------------------------------------------------------------+

Thanks!

@MustaphaYacine

This comment has been minimized.

Show comment
Hide comment
@MustaphaYacine

MustaphaYacine Feb 10, 2018

I'm having the same issue like @j5207 but with CUDA 9.1
Thanks for helping.

MustaphaYacine commented Feb 10, 2018

I'm having the same issue like @j5207 but with CUDA 9.1
Thanks for helping.

@markddesimone

This comment has been minimized.

Show comment
Hide comment
@markddesimone

markddesimone Feb 14, 2018

I have the same issue with cuda-9.1

I tried editing ~/.bashrc by adding the following:

DARKNET

export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

but still have the:
ptxas /tmp/tmpxft_000077c1_00000000-15_convolutional_kernels.compute_30.ptx, line 115; error : Call has wrong number of parameters

etc. errors

I also tried updating the makefile thus:
COMMON+= -DGPU -I/usr/local/cuda-9.1/include/
CFLAGS+= -DGPU
LDFLAGS+= -L/usr/local/cuda-9.1/lib64 -lcuda -lcudart -lcublas -lcurand

but again same error

markddesimone commented Feb 14, 2018

I have the same issue with cuda-9.1

I tried editing ~/.bashrc by adding the following:

DARKNET

export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

but still have the:
ptxas /tmp/tmpxft_000077c1_00000000-15_convolutional_kernels.compute_30.ptx, line 115; error : Call has wrong number of parameters

etc. errors

I also tried updating the makefile thus:
COMMON+= -DGPU -I/usr/local/cuda-9.1/include/
CFLAGS+= -DGPU
LDFLAGS+= -L/usr/local/cuda-9.1/lib64 -lcuda -lcudart -lcublas -lcurand

but again same error

@MatejMatula

This comment has been minimized.

Show comment
Hide comment
@MatejMatula

MatejMatula Mar 17, 2018

i have the same error with cuda 9.1

MatejMatula commented Mar 17, 2018

i have the same error with cuda 9.1

@kalanityL

This comment has been minimized.

Show comment
Hide comment
@kalanityL

kalanityL Mar 29, 2018

Had the same issue with cuda-9.0

  • Added the lines to the .bashrc
    export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda9.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  • did source on the .bashrc

Then compilation worked fine.
Thanks @Dahlasam !

kalanityL commented Mar 29, 2018

Had the same issue with cuda-9.0

  • Added the lines to the .bashrc
    export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda9.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  • did source on the .bashrc

Then compilation worked fine.
Thanks @Dahlasam !

@charithforex

This comment has been minimized.

Show comment
Hide comment
@charithforex

charithforex Apr 16, 2018

@kalanityL
Hi It is already set as follows
export PATH=/usr/local/cuda-9.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64:

still i get the following error at the make

/bin/sh: 1: nvcc: not found
Makefile:91: recipe for target 'obj/convolutional_kernels.o' failed
make: *** [obj/convolutional_kernels.o] Error 127

charithforex commented Apr 16, 2018

@kalanityL
Hi It is already set as follows
export PATH=/usr/local/cuda-9.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64:

still i get the following error at the make

/bin/sh: 1: nvcc: not found
Makefile:91: recipe for target 'obj/convolutional_kernels.o' failed
make: *** [obj/convolutional_kernels.o] Error 127

@kalanityL

This comment has been minimized.

Show comment
Hide comment
@kalanityL

kalanityL Apr 16, 2018

if you do cd /usr/local/cuda-9.0/ and ls, what do you see?

kalanityL commented Apr 16, 2018

if you do cd /usr/local/cuda-9.0/ and ls, what do you see?

@charithforex

This comment has been minimized.

Show comment
Hide comment
@charithforex

charithforex Apr 16, 2018

nvidia@tegra-ubuntu:~$ cd /usr/local/cuda-9.0/
@kalanityL
I am running this on nvidia jetson tx2
nvidia@tegra-ubuntu:/usr/local/cuda-9.0$ ls
bin extras lib64 nvml README share tools
doc include LICENSE nvvm samples targets version.txt
nvidia@tegra-ubuntu:/usr/local/cuda-9.0$

charithforex commented Apr 16, 2018

nvidia@tegra-ubuntu:~$ cd /usr/local/cuda-9.0/
@kalanityL
I am running this on nvidia jetson tx2
nvidia@tegra-ubuntu:/usr/local/cuda-9.0$ ls
bin extras lib64 nvml README share tools
doc include LICENSE nvvm samples targets version.txt
nvidia@tegra-ubuntu:/usr/local/cuda-9.0$

@charithforex

This comment has been minimized.

Show comment
Hide comment
@charithforex

charithforex Apr 16, 2018

@kalanityL
was able figure out with following link

https://stackoverflow.com/questions/39287744/ubuntu-16-04-nvidia-toolkit-8-0-rc-darknet-compilation-error-expected-a

by editing the nvcc=/usr/local/cuda-9.0/bin/nvcc
Thanks for your input

charithforex commented Apr 16, 2018

@kalanityL
was able figure out with following link

https://stackoverflow.com/questions/39287744/ubuntu-16-04-nvidia-toolkit-8-0-rc-darknet-compilation-error-expected-a

by editing the nvcc=/usr/local/cuda-9.0/bin/nvcc
Thanks for your input

@EhabMedhat

This comment has been minimized.

Show comment
Hide comment
@EhabMedhat

EhabMedhat Jun 23, 2018

@charithforex
i updated
nvcc=/usr/local/cuda-80/bin/nvcc
and it worked for me
thanks

EhabMedhat commented Jun 23, 2018

@charithforex
i updated
nvcc=/usr/local/cuda-80/bin/nvcc
and it worked for me
thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment