make darknet 'obj/network_kernels.o' failed

[Krogsager]

When I try to build darknet, I get the error:

~& make clean && make -j8

./src/network_kernels.cu(364): warning: variable "l" was declared but never referenced

./src/network_kernels.cu(706): error: identifier "cudaStreamCaptureModeGlobal" is undefined

./src/network_kernels.cu(706): error: too many arguments in function call

2 errors detected in the compilation of "/tmp/tmpxft_00000d35_00000000-6_network_kernels.cpp1.ii".
Makefile:185: recipe for target 'obj/network_kernels.o' failed
make: *** [obj/network_kernels.o] Error 1

Darknet repo is a clone of commit b6cd759.
I've read multiple issues on the matter but they seem to focus on an older version of CUDA.
Notice I'm running on Quadro P6000 so my Makefile looks like this:

GPU=1
CUDNN=1
CUDNN_HALF=0
OPENCV=1
AVX=1
OPENMP=1
LIBSO=0
ZED_CAMERA=0
ZED_CAMERA_v2_8=0

USE_CPP=0
DEBUG=0

ARCH= -gencode arch=compute_35,code=sm_35 \
      -gencode arch=compute_50,code=[sm_50,compute_50] \
      -gencode arch=compute_52,code=[sm_52,compute_52] \
	    -gencode arch=compute_61,code=[sm_61,compute_61]

OS := $(shell uname)

# GeForce RTX 2080 Ti, RTX 2080, RTX 2070, Quadro RTX 8000, Quadro RTX 6000, Quadro RTX 5000, Tesla T4, XNOR Tensor Cores
ARCH= -gencode arch=compute_75,code=[sm_75,compute_75]

My system setup:

Ubuntu 18.04.5 LST

~$ nvidia-smi
Tue Apr 20 05:23:00 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.56       Driver Version: 460.56       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P6000        Off  | 00000000:00:05.0  On |                 ERR! |
| 26%   36C    P8    14W / 250W |    224MiB / 24449MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1481      G   /usr/lib/xorg/Xorg                 97MiB |
|    0   N/A  N/A      1947      G   /usr/bin/gnome-shell              123MiB |
+-----------------------------------------------------------------------------+

installed drivers

~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  460.56  Tue Feb 23 23:31:36 UTC 2021
GCC version:  gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) 

nvcc version

~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

gcc version

~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)

[Krogsager]

I changed my brain from stupid -> minimal effort and found a fix: #7557 (comment)
As aldisio shows in his post, checkout version 64efa72 fixes the issue.
I will leave the issue open for now.

[haviduck]

had this issue today on ubuntu 18.04. relevant fix for me was setting nvcc=cudalib64path/bin in makefile. additionally changed opencv4 to opencv in pkg-config lines. ran sudo make successfully after that

[stephanecharette]

ran sudo make successfully after that

Why run *sudo* make? Won't that mess up your build folder by leaving a bunch of files behind owned by root?

[haviduck]

ran sudo make successfully after that

Why run *sudo* make? Won't that mess up your build folder by leaving a bunch of files behind owned by root?

indeed! listen to this one :)

[Krogsager]

had this issue today on ubuntu 18.04. relevant fix for me was setting nvcc=cudalib64path/bin in makefile. additionally changed opencv4 to opencv in pkg-config lines. ran sudo make successfully after that

Hi haviduck, could you show me what you changed? I'm new to cuda and compiling from source, so its all a bit overwhelming :)
When I search my machine I only find a single file similar to cudalib.

$ find / -type f -iname "*cudalib*" 2>/dev/null
/usr/local/cuda-10.0/targets/x86_64-linux/include/cudalibxt.h

[haviduck]

had this issue today on ubuntu 18.04. relevant fix for me was setting nvcc=cudalib64path/bin in makefile. additionally changed opencv4 to opencv in pkg-config lines. ran sudo make successfully after that

Hi haviduck, could you show me what you changed? I'm new to cuda and compiling from source, so its all a bit overwhelming :)
When I search my machine I only find a single file similar to cudalib.

$ find / -type f -iname "*cudalib*" 2>/dev/null
/usr/local/cuda-10.0/targets/x86_64-linux/include/cudalibxt.h

my bad, it was a rushed reply and wrong aswell. i dont know what os you are on, so might differ, but heres my path NVCC=/usr/local/cuda-11.2/bin/nvcc

[Krogsager]

The issue of 'obj/network_kernels.o' failed was somehow related to my Nvidia driver and cuda toolkit installation. I can only recommend people to follow the install instructions very carefully, both pre and post installation.
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions

Also, take a look at what you have installed on your system. Got more than version of cuda perhaps?

 sudo apt list --installed | grep cuda

This solved my issue
https://askubuntu.com/questions/140246/how-do-i-resolve-unmet-dependencies-after-adding-a-ppa