Build using nvcc + clang?

[fm966mhz]

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
• TensorFlow installed from (source or binary): source
• TensorFlow version: commit c044803 (with the problem in Build fails with messages including "undefined reference to symbol 'acos@@GLIBC_2.2.5' ", "Linking of rule '@flatbuffers//:flatc' failed" #36170 fixed locally)
• Python version: 3.7.6
• Installed using virtualenv? pip? conda?: conda
• Bazel version (if compiling from source): 1.2.1
• GCC/Compiler version (if compiling from source): gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
• CUDA/cuDNN version: V10.1.243/7.6.5
• GPU model and memory: GeForce GTX 1080, 8116 MiB

Describe the problem

I was able to build Tensorflow with (bazel build tensorflow/tools/pip_package:build_pip_package) using nvcc + gcc. But if I tried to manully override gcc using clang (8.0.1), the build will fail with

clang: error: unknown argument: '-fno-canonical-system-headers'

It seems the CC toolchain auto config still generates flags for nvcc + gcc, even though I choose to use clang. I confirmed that my gcc has this flag while my clang does not.

If I choose to use clang as CUDA compiler, it would fail with

 error: cannot find libdevice for sm_61.

which seems to mean that Clang cannot compile for this cuda compute capabilities (6.1). So I am wondering if there is any way of using nvcc + clang to build since I use clang as default (for all other builds). Shouldn't this be possible since clang is used by default on Macs? Thanks!

[fm966mhz]

The original error when using clang as cuda compiler (cannot find libdevice) seems to go away after upgrading clang to a new version (10.0.0-++20200126090219+73a91477f70-1exp120200126201656.19). Is the build supposed to be done either by gcc+nvcc or purely by clang? Or is Clang supported?

Using the new clang version still fails due to errors like the following. To isolate the problem, the target I built is

bazel build @nccl_archive//:device_lib

The errors look like this

In file included from <built-in>:1:
In file included from /usr/lib/llvm-10/lib/clang/10.0.0/include/__clang_cuda_runtime_wrapper.h:202:
In file included from /usr/include/string.h:494:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:95:8: error: exception specification in declaration does not match previous declaration
__NTH (stpcpy (char *__restrict __dest, const char *__restrict __src))
       ^
/usr/include/string.h:451:14: note: previous declaration is here
extern char *stpcpy (char *__restrict __dest, const char *__restrict __src)
             ^

There is one such error for each function in /usr/include/string.h (installed from libc6-dev 2.27-3). It seems to be caused by the difference in their declaration and definition. gcc doesn't take it as an error while clang-10 does. For example, the declaration of strcpy in string.h is

/* Copy SRC to DEST.  */
extern char *strcpy (char *__restrict __dest, const char *__restrict __src)
     __THROW __nonnull ((1, 2));

while in /usr/include/x86_64-linux-gnu/bits/string_fortified.h it is

__fortify_function char *
__NTH (stpcpy (char *__restrict __dest, const char *__restrict __src))
{
  return __builtin___stpcpy_chk (__dest, __src, __bos (__dest));
}

The latter does not throw while the former does? I am not sure if this should be a Tensorflow issue or Clang issue or there is something wrong with my set up.

[fm966mhz]

I also saw the same error came up in this issue: #34098

[gunan]

@Artem-B may have some details.
At one point, I remember CUDA headers had a check to block building with Clang, when we first tried this. I do not know if this still applies.

Another option is to just use clang when building TF. Clang can already compile cuda code.

[Artem-B]

NVCC + custom crosstool w/ a wrapper script was very gcc-centric when I looked at it last time (mind, it was a while back). I would not expect that to work out of the box.

As for using clang for CUDA compilation, it's expected to work in general, but I don't have enough details to tell what went wrong in this case.

error: cannot find libdevice for sm_61.

This could happen if you install CUDA from packages (as opposed to installing it manually from a .run file). Packages tend to scatter CUDA files all over the place which confuses clang. I think there were recent changes to attempt dealing with some of this, but in general clang assumes a monolithic CUDA installation (I.e. all CUDA files are under a single directory).

As for mismatching declarations for stpcpy, here's how I'd debug it:

• run blaze with --verbose_failures and extract exact compiler command line
• reproduce the failure manually
• If the failure was during compilation of a CUDA file run compiler with -v and extract command line of sub-compilation that fails.
• run the failing sub-compilation with -dD -E and check what preprocessed declarations/definitions look like for the functions we're failing on.
• track down the differences to a macro which leads to them (__USE_XOPEN2K8 ? __THROW ?) and then figure out why the declarations are inconsistent.

One thing that could be the culprit is that CUDA does not support exceptions on the GPU side. If something/somewhere assumes that exceptions are available, things may not work as expected. Another possible source of trouble is that clang's __clang_cuda_runtime_wrapper.h which implements the magic massaging CUDA headers into something clang can pasre may need to be updated to deal with more recent headers on your machine.

@djwenren If you can send me exact command line that fails, I'll try reproducing it locally and check what's going on. The headers on my machine appear to match the ones quoted above.

[fm966mhz]

Thanks for the detailed information, @Artem-B ! I reinstalled my Cuda toolkit (10.1) using runfiles from Nvidia. The problems seem to have gone away. I was able to build using purely Clang. Could have been some misconfiguration I had earlier.

[tensorflow-bot]

Are you satisfied with the resolution of your issue?
Yes
No