cuda/cuda_config.h missing when compiling custom ops with nvcc #12860

pronobis · 2017-09-07T00:12:05Z

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Yes, see below.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Ubuntu 16.04
TensorFlow installed from (source or binary):
Source
TensorFlow version (use command below):
v1.3.0-0-g9e76bf324 1.3.0
Python version:
3.5.2
Bazel version (if compiling from source):
0.5.4
CUDA/cuDNN version:
8.0.44 / 5.1.5
GPU model and memory:
Any.
Exact command to reproduce:
See below.

Describe the problem

When compiling a custom op using nvcc, which includes tensorflow/core/util/cuda_kernel_helper.h, I get the following error:

/usr/local/cuda-8.0/bin/nvcc -c -o ~/Code/libspn/build/ops/gather_columns_functor_gpu.cu.cc.o ~/Code/libspn/libspn/ops/gather_columns_functor_gpu.cu.cc -std=c++11 -x=cu -Xcompiler -fPIC -DGOOGLE_CUDA=1 --expt-relaxed-constexpr -I ~/.local/lib/python3.5/site-packages/tensorflow/include -gencode=arch=compute_35,"code=sm_35,compute_35" -gencode=arch=compute_52,"code=sm_52,compute_52" -gencode=arch=compute_61,"code=sm_61,compute_61"
In file included from ~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/core/platform/default/stream_executor.h:26:0,
                 from ~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/core/platform/stream_executor.h:24,
                 from ~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:26,
                 from ~/Code/libspn/libspn/ops/gather_columns_functor_gpu.cu.h:11,
                 from ~/Code/libspn/libspn/ops/gather_columns_functor_gpu.cu.cc:5:
~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/stream_executor/dso_loader.h:32:30: fatal error: cuda/cuda_config.h: No such file or directory
compilation terminated.

Copying cuda_config.h to /site-packages/tensorflow/include/tensorflow/stream_executor/cuda solves the problem.

The same issue has been observed by several other users in #6602 (see the comments added after the issue was closed).

The text was updated successfully, but these errors were encountered:

yangyu12 · 2017-09-07T06:34:24Z

the same problem, I want to know if there is any better way to solve this problem

aselle · 2017-09-20T20:32:22Z

@allenlavoie, have you looked at this as part of your library cleanup?

allenlavoie · 2017-09-22T00:39:18Z

I haven't. I may have run into it when I was following our custom op documentation. It sounds like worst case we could add a copy in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/pip_package/build_pip_package.sh ? Or maybe a genrule before that.

jmlago · 2017-09-28T10:26:06Z

I have the same problem while compiling with nvcc. Also is because tensorflow/core/util/cuda_kernel_helper.h was included.

But when I copy the file, i get the following error:

../site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:28:80: fatal error: cuda/include/cuda.h: No such file or directory
compilation terminated.

PatWie · 2017-10-11T00:12:54Z

The commit 2c598e8 destroys my makefile, as well. This commit consistly changes in several places the includes

-#include "third_party/gpus/cuda/include/cuda.h"
+#include "cuda/include/cuda.h"

Which gives me the error message:

../site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:28:80: fatal error: cuda/include/cuda.h: No such file or directory
compilation terminated.

compiling user-ops with GPU does not work since 1.3.1.

Where lives that guy, tensorflower-gardener? He is the one who usually introduces those breaking changes!

HelloSeeing · 2017-10-17T17:25:22Z

@PatWie +1
I encounter this problem when build android tensorflow_demo, giving the following error message:

tensorflow/core/kernels/lrn_op.cc:34:10: fatal error: 'cuda/include/cuda.h' file not found

pronobis · 2017-10-24T23:44:38Z

With TF 1.3, I only find cuda_config.h missing, but when compiling against current master 1.4-rc0/1, I also additionally get:

tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:24:31: fatal error: cuda/include/cuda.h: No such file or directory
compilation terminated.

bigbigda · 2017-11-01T13:47:29Z

I also encounter the same problem in TF 1.3.0. This error is introduced with #include "third_party/eigen3/unsupported/Eigen/CXX11/Tensor"
/usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/stream_executor/dso_loader.h:32:30: fatal error: cuda/cuda_config.h: No such file or directory

Leo-Mrl · 2017-11-03T14:38:07Z

I also encounter this problem in TF 1.4.0, any clue ?

kkk324 · 2017-11-10T08:04:36Z

Also stucking on this problem in TF 1.4.0...

Leo-Mrl · 2017-11-10T09:35:11Z

@CR-Ko Downgrading to 1.2.0 "solved" the issue for me

nikste · 2017-11-13T17:08:34Z

I have a similar issue, resulting in:
.tensorflow-venv/lib/python3.6/site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:24:31: fatal error: cuda/include/cuda.h: No such file or directory
Any other workarounds for this besides downgrading?

Queequeg92 · 2017-11-17T00:34:11Z

Downgrading "solution" will cause incompatibility with cudnn. You also need to downgrade cudnn.@nikste
Could installing from source solve this problem?@pronobis @PatWie

PatWie · 2017-11-17T10:28:24Z

No change here. I currently stick to TF1.2 compiled from source (I always compile the library from source). It is frustrating. @allenlavoie or @aselle can you give us at least a hint?

edit: I tracked it down to commit 2c598e8.

This is a MWE for v1.2 for reproducing the error. It works under TF1.2 but not TF1.4.

nikste · 2017-11-27T08:00:36Z

@Queequeg92 installing from source did not fix the problem, ended up manually adding paths to .h files as additional include directories and copying cuda_config.h from source to my project.

tensorflowbutler · 2017-12-20T01:09:09Z

It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

John1231983 · 2018-01-03T18:42:37Z

Hello. I also have the problem with T.F 1.4 although it worked in T.F 1.2. Have anyone fix it with T.F 1.4

gunan · 2018-01-05T22:02:39Z

Folding all the duplicates into #15002

PatWie · 2018-01-09T13:22:12Z

@Queequeg92 I have it working under v1.5rc0 now, see https://github.com/cgtuebingen/tf_custom_op which requires installing from source or copying the cuda_config.h from #15002.

Queequeg92 · 2018-01-10T01:37:22Z

@PatWie Thank you! I'll have a try.

Queequeg92 · 2018-01-18T12:19:57Z

@PatWie I have installed v1.5 from source successfully. No cuda_config.h error, but still cuda.h error. 😫

xysmlx · 2018-02-06T08:58:26Z

I solved this problem by commenting the line *** #include "cuda/cuda_config.h" *** in the "dso_loader.h" file. And the custom op works normally.

PatWie · 2018-02-06T12:37:20Z

@Queequeg92
The cuda.h is part of the cuda toolkit or whatever Nvidia ships. It is not part of TensorFlow, that is what the lines

# use cuda
find_package(CUDA 9.0 EXACT REQUIRED)
set(CUDA_SAMPLE_INC "${CUDA_INCLUDE_DIRS}/../samples/common/inc")
message(STATUS "CUDA_INCLUDE_DIRS: ${CUDA_INCLUDE_DIRS}")
include_directories(SYSTEM "${CUDA_INCLUDE_DIRS}/../../")

are for. But the latest version of TF is broken again (#15002 (comment))

@xysmlx
When writing cuda ops, you probably want to use the TensorFlow cuda_config part.

cy89 mentioned this issue Sep 14, 2017

fatal error: tensorflow/stream_executor/lib/status.h: No such file or directory #6602

Closed

cy89 added the type:build/install Build and install issues label Sep 15, 2017

aselle added stat:awaiting response Status - Awaiting response from author stat:awaiting tensorflower Status - Awaiting response from tensorflower and removed stat:awaiting response Status - Awaiting response from author labels Sep 20, 2017

PatWie mentioned this issue Nov 30, 2017

fatal error: cuda/include/cuda.h: No such file or directory #15002

Closed

Connor323 mentioned this issue Nov 30, 2017

sparse input Connor323/Convolution-with-sparse-kernel-in-TF#2

Closed

gunan closed this as completed Jan 5, 2018

gunan mentioned this issue Feb 12, 2018

Remove header dependence on cuda_config.h to fix opensource custom op… #16959

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cuda/cuda_config.h missing when compiling custom ops with nvcc #12860

cuda/cuda_config.h missing when compiling custom ops with nvcc #12860

pronobis commented Sep 7, 2017

yangyu12 commented Sep 7, 2017

aselle commented Sep 20, 2017

allenlavoie commented Sep 22, 2017

jmlago commented Sep 28, 2017 •

edited

PatWie commented Oct 11, 2017 •

edited

HelloSeeing commented Oct 17, 2017

pronobis commented Oct 24, 2017

bigbigda commented Nov 1, 2017 •

edited

Leo-Mrl commented Nov 3, 2017

kkk324 commented Nov 10, 2017

Leo-Mrl commented Nov 10, 2017

nikste commented Nov 13, 2017 •

edited

Queequeg92 commented Nov 17, 2017 •

edited

PatWie commented Nov 17, 2017 •

edited

nikste commented Nov 27, 2017 •

edited

tensorflowbutler commented Dec 20, 2017

John1231983 commented Jan 3, 2018

gunan commented Jan 5, 2018

PatWie commented Jan 9, 2018

Queequeg92 commented Jan 10, 2018

Queequeg92 commented Jan 18, 2018

xysmlx commented Feb 6, 2018

PatWie commented Feb 6, 2018 •

edited

cuda/cuda_config.h missing when compiling custom ops with nvcc #12860

cuda/cuda_config.h missing when compiling custom ops with nvcc #12860

Comments

pronobis commented Sep 7, 2017

System information

Describe the problem

yangyu12 commented Sep 7, 2017

aselle commented Sep 20, 2017

allenlavoie commented Sep 22, 2017

jmlago commented Sep 28, 2017 • edited

PatWie commented Oct 11, 2017 • edited

HelloSeeing commented Oct 17, 2017

pronobis commented Oct 24, 2017

bigbigda commented Nov 1, 2017 • edited

Leo-Mrl commented Nov 3, 2017

kkk324 commented Nov 10, 2017

Leo-Mrl commented Nov 10, 2017

nikste commented Nov 13, 2017 • edited

Queequeg92 commented Nov 17, 2017 • edited

PatWie commented Nov 17, 2017 • edited

nikste commented Nov 27, 2017 • edited

tensorflowbutler commented Dec 20, 2017

John1231983 commented Jan 3, 2018

gunan commented Jan 5, 2018

PatWie commented Jan 9, 2018

Queequeg92 commented Jan 10, 2018

Queequeg92 commented Jan 18, 2018

xysmlx commented Feb 6, 2018

PatWie commented Feb 6, 2018 • edited

jmlago commented Sep 28, 2017 •

edited

PatWie commented Oct 11, 2017 •

edited

bigbigda commented Nov 1, 2017 •

edited

nikste commented Nov 13, 2017 •

edited

Queequeg92 commented Nov 17, 2017 •

edited

PatWie commented Nov 17, 2017 •

edited

nikste commented Nov 27, 2017 •

edited

PatWie commented Feb 6, 2018 •

edited