New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile fails using g++ 4.8 with "kernel launches from templates are not allowed in system files" #337

Closed
AndrewLavin opened this Issue Apr 20, 2014 · 25 comments

Comments

Projects
None yet
@AndrewLavin

AndrewLavin commented Apr 20, 2014

Build fails using g++ 4.8 with error message "kernel launches from templates are not allowed in system files". Using g++ 4.6 instead works.

Here is the compiler error.

/opt/nvidia/cuda/bin/nvcc -ccbin=/usr/bin/g++ -Xcompiler -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./include -I/opt/nvidia/cuda/include -I/opt/intel/mkl/include -gencode arch=compute_20,code=sm_20 -gencode arch=compute_20,code=sm_21 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -c src/caffe/layers/pooling_layer.cu -o build/src/caffe/layers/pooling_layer.cuo
src/caffe/layers/pooling_layer.cu(164): error: kernel launches from templates are not allowed in system files

src/caffe/layers/pooling_layer.cu(171): error: kernel launches from templates are not allowed in system files

src/caffe/layers/pooling_layer.cu(291): error: kernel launches from templates are not allowed in system files

src/caffe/layers/pooling_layer.cu(298): error: kernel launches from templates are not allowed in system files

src/caffe/layers/pooling_layer.cu(305): error: kernel launches from templates are not allowed in system files

5 errors detected in the compilation of "/tmp/tmpxft_00006a5e_00000000-10_pooling_layer.cpp4.ii".
make: *** [build/src/caffe/layers/pooling_layer.cuo] Error 2

@shiquanwang

This comment has been minimized.

shiquanwang commented Apr 28, 2014

I met the same issue.

sqwang@sqwang-Z820:~/workspace/project/open-source/caffe$ make
/usr/local/cuda/bin/nvcc -ccbin=/usr/bin/g++ -Xcompiler -fPIC -DNDEBUG -O2 -I/home/sqwang/anaconda/include -I/home/sqwang/anaconda/include/python2.7 -I/home/sqwang/anaconda/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/opt/opencv/opencv-2.4.9/include -Ibuild/src -I./src -I./include -I/usr/local/cuda/include -gencode arch=compute_20,code=sm_20 -gencode arch=compute_20,code=sm_21 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -c src/caffe/layers/softmax_layer.cu -o build/src/caffe/layers/softmax_layer.cuo
src/caffe/layers/softmax_layer.cu(59): error: kernel launches from templates are not allowed in system files

src/caffe/layers/softmax_layer.cu(66): error: kernel launches from templates are not allowed in system files

src/caffe/layers/softmax_layer.cu(73): error: kernel launches from templates are not allowed in system files

3 errors detected in the compilation of "/tmp/tmpxft_00006d79_00000000-10_softmax_layer.cpp4.ii".
@Yangqing

This comment has been minimized.

Member

Yangqing commented Apr 28, 2014

Possible duplicate of #27

@kloudkl

This comment has been minimized.

Contributor

kloudkl commented May 21, 2014

The trick that I used to fix the similar problem in #27 on Ubuntu 12.04(.*) does not work on Ubuntu 14.04 whose gcc/g++ version is 4.8+. The only way is to use an older compiler that CUDA supports.

sudo apt-get install gcc-4.6 g++-4.6 gcc-4.6-multilib g++-4.6-multilib
sudo update-alternatives --remove-all gcc
sudo update-alternatives --remove-all g++
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 20
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.6 20
sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30
sudo update-alternatives --set cc /usr/bin/gcc
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30
sudo update-alternatives --set c++ /usr/bin/g++
sudo update-alternatives --config gcc
sudo update-alternatives --config g++
@weinman

This comment has been minimized.

Contributor

weinman commented May 23, 2014

I compiled (successfully passing all tests and running the MNIST example) on Ubuntu 14.04 by simply installing gcc/g++ 4.6 (first apt-get install by kloudkl above) and then changing the Makefile to point to the appropriate compiler:

CXX := /usr/bin/g++-4.6

This way you don't have to globally set your alternatives to a retrograde gcc/g++ version.

@AlexBinder

This comment has been minimized.

AlexBinder commented May 30, 2014

CXX := /usr/bin/g++-4.6 does not work with the current Makefile bcs the compiler choice is hardcoded right now in Makefile, see Line 155

ifeq ($(LINUX), 1)
CXX := /usr/bin/g++
endif

so you need to change this in the makefile right now (to g++-4.6 as weinman said ) ... anybody wants to open a minor issue, so that CXX is used from Makefile.config again ?

@escorciav

This comment has been minimized.

escorciav commented Jun 17, 2014

@AlexBinder if you agree and you didn't submit it, I will submit this minor issue. I consider that it is important that people choose gcc compiler due to previous errors.
It is my third attempt to compile Caffe and I got this error again XD!!! Fortunately and thanks to caffe community, I can overcome it.

@yrevar

This comment has been minimized.

Contributor

yrevar commented Jun 17, 2014

Can be fixed easily by modifying it as,
CXX ?= /usr/bin/g++

@escorciav

This comment has been minimized.

escorciav commented Jun 17, 2014

I agree. I modified two lines of my Makefile as you mention and I include CXX in Makefile.config. As Alex said, it is just a trivial issue but it could help rookie linux users.

@AlexBinder

This comment has been minimized.

AlexBinder commented Jun 18, 2014

@escorciav I think it is a good idea that you fixed that. I just did not wanted to open an issue on that or change compilation settings when I was not sure whether there was some reason which I was unaware of.

@escorciav

This comment has been minimized.

escorciav commented Jun 18, 2014

Ok, it has sense and I didn't think about that.

@shelhamer

This comment has been minimized.

Member

shelhamer commented Jul 4, 2014

#611 fixes the issue with setting a custom compiler raised and discussed by @AlexBinder @escorciav @yrevar

@shelhamer

This comment has been minimized.

Member

shelhamer commented Jul 4, 2014

@AndrewLavin @shiquanwang @kloudkl

I compiled and tested Caffe with

  • Ubuntu 14.04
  • CUDA 6.0.37
  • driver 331.75
  • g++ 4.8

and did not encounter the "kernel launches from templates are not allowed in system files" error. However, Caffe performance was poor as noted elsewhere (backward pass was 3.4x slower than normal).

@AlexBinder

This comment has been minimized.

AlexBinder commented Jul 4, 2014

Hi Evan,

thanks for the report. Noted ... to solve this problem one should build Cuda from source (rather than using the ubuntu package which is as of today is still only 5.5) so that it works with g++-4.8 .

The default which ubuntu 14.04 offers today is
apt search nvidia-cuda-toolkit
Sorting... Done
Full Text Search... Done
nvidia-cuda-toolkit/trusty,now 5.5.22-3ubuntu1 amd64 [installed]
NVIDIA CUDA toolkit
With that older version it produces with g++4.8 still the kernel launch problem.

That should explain the difference between your result and the result of the others. Best, Alex

@kloudkl

This comment has been minimized.

Contributor

kloudkl commented Jul 4, 2014

I think that g++4.8 is still somewhat experimental.

@escorciav

This comment has been minimized.

escorciav commented Jul 5, 2014

Thank you shelmaher. I have not tested neither speed nor CUDA 6.0 standard release but it is sure that compiler var could help as much as include and library var.
I recommend to check which is the compiler suggested for your CUDA version and to use the latest driver, if it is possible. If you are using Fedora (active development), it is extremely useful.

@rmanor

This comment has been minimized.

Contributor

rmanor commented Aug 9, 2014

I also encountered this problem after upgrading to Ubuntu 13.10 .
Reverting to g++ 4.7 solved the problem.
This should be noted in the installation documentation.

@sachinky

This comment has been minimized.

sachinky commented Aug 23, 2014

I am also getting this error when I tried compiling caffe after compiling MATLAB wrapper.

To compile matlab wrapper I used the following command instead of 'make matcaffe'
g++ -shared -fPIC matcaffe.cpp -o matcaffe

because I was getting the following error with 'make matcaffe'
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/crt1.o: In function _start': (.text+0x20): undefined reference tomain'

I did not get any error when compiled caffe without using matlab wrapper.I succesfully passed all the test cases.

Kindly help ASAP.

@kloudkl

This comment has been minimized.

Contributor

kloudkl commented Aug 27, 2014

@alexbw

This comment has been minimized.

alexbw commented Sep 13, 2014

Would be great to have this information available on the installation page. http://caffe.berkeleyvision.org/installation.html

@en4bz

This comment has been minimized.

en4bz commented Oct 26, 2014

I avoided this by setting CUSTOM_CXX := g++-4.7 in the Makefile.config.

ubuntu 14.10
cuda 5.5
arch = compute_30
g++ = 4.7
driver 331.89

@magicknight

This comment has been minimized.

magicknight commented Jan 9, 2015

I got the same error and I switch to gcc 4.6 then it works

@RyanCarey

This comment has been minimized.

RyanCarey commented May 18, 2015

It also corrects for me when switching to g++-4.6

@fuglede

This comment has been minimized.

fuglede commented Jul 9, 2015

Ran into the same problem using the g++-4.9 shipped with Debian Jessie. Downgrading to g++-4.6 from testing and using CUSTOM_CXX did the trick here as well.

@Jack-Foster

This comment has been minimized.

Jack-Foster commented Nov 4, 2015

Hi, I got the same problem using g++4.8 & 4.9 with Debian Jessie. I tried using g++-4.6 but it doesn't work neither because the recent version of caffe requests some new includes that do not exist for g++4.6.
For instance in .build_release/src/caffe/proto/caffe.pb.h line 7 there is :
#include

This include exists for g++-4.8 but doesn't for g++-4.6....
I also tried to use the include from g++-4.8 using g++-4.6 for compilation but it (obviously) gives weird errors :

In file included from /usr/include/c++/4.8/bits/basic_string.h:39:0,
from /usr/include/c++/4.8/string:52,
from /usr/include/google/protobuf/stubs/common.h:41,
from .build_release/src/caffe/proto/caffe.pb.h:9,
from .build_release/src/caffe/proto/caffe.pb.cc:5:
/usr/include/c++/4.8/ext/atomicity.h: In function ‘_Atomic_word __gnu_cxx::__exchange_and_add(volatile Atomic_word, int)’:
/usr/include/c++/4.8/ext/atomicity.h:49:45: error: ‘__ATOMIC_ACQ_REL’ was not declared in this scope
/usr/include/c++/4.8/ext/atomicity.h:49:61: error: ‘__atomic_fetch_add’ was not declared in this scope
/usr/include/c++/4.8/ext/atomicity.h: In function ‘void __gnu_cxx::__atomic_add(volatile Atomic_word, int)’:
/usr/include/c++/4.8/ext/atomicity.h:53:38: error: ‘__ATOMIC_ACQ_REL’ was not declared in this scope
/usr/include/c++/4.8/ext/atomicity.h:53:54: error: ‘__atomic_fetch_add’ was not declared in this scope
Makefile:551: recipe for target '.build_release/src/caffe/proto/caffe.pb.o' failed

Could anyone give me some help on this ? Thx

@dgeo

This comment has been minimized.

dgeo commented Nov 18, 2015

Tried here on debian jessie to install using nvidia-cuda-toolkit-6.5.14 from stretch instead of using gcc-4.6 from wheezy: seems to works for me™ here with a K80.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment