[HIPIFY] Hipfy-clang does not support Thrust APIs shipped with CUDA 8.0

[b8875]

I got the following error when I translated Deepbench cuda codes that contain thurst APIs.
/usr/local/cuda-8.0/include/thrust/detail/swap.h:25:1: error: expected unqualified-id
thrust_exec_check_disable

[emankov]

Could you please provide more information. 1. OS, 2. Clang version, 3. the whole command line, 3. the whole output log. Thanks.

[b8875]

OS: 16.04.4
Clang Version: LLVM version 4.0.0
Command: hipify-clang -print-stats interac.cu -- -x cuda --cuda-path=/usr/local/cuda-8.0 --cuda-gpu-arch=sm_50

/usr/local/cuda-8.0/include/thrust/iterator/iterator_adaptor.h:187:5: error: expected member name or ';' after declaration specifiers
thrust_exec_check_disable
^
/usr/local/cuda-8.0/include/thrust/detail/config/exec_check_disable.h:29:43: note: expanded from macro 'thrust_exec_check_disable'

define thrust_exec_check_disable #pragma hd_warning_disable

                                      ^

In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:24:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pointer.h:20:
/usr/local/cuda-8.0/include/thrust/iterator/iterator_adaptor.h:149:9: error: member initializer 'm_iterator' does not name a non-static data
member or base class
: m_iterator(iter)
^~~~~~~~~~~~~~~~
/usr/local/cuda-8.0/include/thrust/iterator/iterator_adaptor.h:168:14: error: use of undeclared identifier 'm_iterator'
{ return m_iterator; }
^
/usr/local/cuda-8.0/include/thrust/iterator/iterator_adaptor.h:175:14: error: use of undeclared identifier 'm_iterator'
{ return m_iterator; }
^
/usr/local/cuda-8.0/include/thrust/iterator/iterator_adaptor.h:181:14: error: use of undeclared identifier 'm_iterator'
{ return m_iterator; }
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:24:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pointer.h:192:
/usr/local/cuda-8.0/include/thrust/detail/pointer.inl:27:7: error: host function 'pointer<Element, Tag, Reference, Derived>' cannot
overload host device function 'pointer<Element, Tag, Reference, Derived>'
::pointer()
^
/usr/local/cuda-8.0/include/thrust/detail/pointer.h:146:5: note: previous declaration is here
pointer();
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:24:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pointer.h:192:
/usr/local/cuda-8.0/include/thrust/detail/pointer.inl:35:9: error: host function 'pointer<Element, Tag, Reference, Derived>' cannot
overload host device function 'pointer<Element, Tag, Reference, Derived>'
::pointer(OtherElement *other)
^
/usr/local/cuda-8.0/include/thrust/detail/pointer.h:152:14: note: previous declaration is here
explicit pointer(OtherElement *ptr);
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:24:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pointer.h:192:
/usr/local/cuda-8.0/include/thrust/detail/pointer.inl:43:9: error: host function 'pointer<Element, Tag, Reference, Derived>' cannot
overload host device function 'pointer<Element, Tag, Reference, Derived>'
::pointer(const OtherPointer &other,
^
/usr/local/cuda-8.0/include/thrust/detail/pointer.h:158:5: note: previous declaration is here
pointer(const OtherPointer &other,
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:24:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pointer.h:192:
/usr/local/cuda-8.0/include/thrust/detail/pointer.inl:60:11: error: host function 'operator=' cannot overload host device function
'operator='
::operator=(const OtherPointer &other)
^
/usr/local/cuda-8.0/include/thrust/detail/pointer.h:175:5: note: previous declaration is here
operator=(const OtherPointer &other);
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:24:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pointer.h:192:
/usr/local/cuda-8.0/include/thrust/detail/pointer.inl:70:9: error: host function 'dereference' cannot overload host device function
'dereference'
::dereference() const
^
/usr/local/cuda-8.0/include/thrust/detail/pointer.h:134:33: note: previous declaration is here
typename super_t::reference dereference() const;
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:24:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pointer.h:192:
/usr/local/cuda-8.0/include/thrust/detail/pointer.inl:78:7: error: host function 'get' cannot overload host device function 'get'
::get() const
^
/usr/local/cuda-8.0/include/thrust/detail/pointer.h:180:14: note: previous declaration is here
Element *get() const;
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:30:
In file included from /usr/local/cuda-8.0/include/thrust/pair.h:282:
In file included from /usr/local/cuda-8.0/include/thrust/detail/pair.inl:18:
/usr/local/cuda-8.0/include/thrust/detail/swap.h:25:1: error: expected unqualified-id
thrust_exec_check_disable
^
/usr/local/cuda-8.0/include/thrust/detail/config/exec_check_disable.h:29:43: note: expanded from macro 'thrust_exec_check_disable'

#define thrust_exec_check_disable #pragma hd_warning_disable
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:30:
In file included from /usr/local/cuda-8.0/include/thrust/pair.h:282:
/usr/local/cuda-8.0/include/thrust/detail/pair.inl:25:7: error: host function 'pair<T1, T2>' cannot overload host device function
'pair<T1, T2>'
::pair(void)
^
/usr/local/cuda-8.0/include/thrust/pair.h:71:23: note: previous declaration is here
host device pair(void);
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:30:
In file included from /usr/local/cuda-8.0/include/thrust/pair.h:282:
/usr/local/cuda-8.0/include/thrust/detail/pair.inl:34:7: error: host function 'pair<T1, T2>' cannot overload host device function
'pair<T1, T2>'
::pair(const T1 &x, const T2 &y)
^
/usr/local/cuda-8.0/include/thrust/pair.h:79:3: note: previous declaration is here
pair(const T1 &x, const T2 &y);
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:30:
In file included from /usr/local/cuda-8.0/include/thrust/pair.h:282:
/usr/local/cuda-8.0/include/thrust/detail/pair.inl:44:9: error: host function 'pair<T1, T2>' cannot overload host device function
'pair<T1, T2>'
::pair(const pair<U1,U2> &p)
^
/usr/local/cuda-8.0/include/thrust/pair.h:92:3: note: previous declaration is here
pair(const pair<U1,U2> &p);
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:30:
In file included from /usr/local/cuda-8.0/include/thrust/pair.h:282:
/usr/local/cuda-8.0/include/thrust/detail/pair.inl:54:9: error: host function 'pair<T1, T2>' cannot overload host device function
'pair<T1, T2>'
::pair(const std::pair<U1,U2> &p)
^
/usr/local/cuda-8.0/include/thrust/pair.h:105:3: note: previous declaration is here
pair(const std::pair<U1,U2> &p);
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:71:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.inl:21:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/adl/malloc_and_free.h:32:
In file included from /usr/local/cuda-8.0/include/thrust/system/cuda/detail/malloc_and_free.h:23:
In file included from /usr/local/cuda-8.0/include/thrust/system/system_error.h:29:
In file included from /usr/local/cuda-8.0/include/thrust/system/error_code.h:518:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/error_category.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/functional.h:26:
In file included from /usr/local/cuda-8.0/include/thrust/detail/functional/placeholder.h:20:
In file included from /usr/local/cuda-8.0/include/thrust/detail/functional/actor.h:29:
In file included from /usr/local/cuda-8.0/include/thrust/tuple.h:34:
/usr/local/cuda-8.0/include/thrust/detail/tuple.inl:813:13: error: no function template matches function template specialization 'eq'
inline bool eq<null_type,null_type>(const null_type&, const null_type&) { return true; }
^
/usr/local/cuda-8.0/include/thrust/detail/tuple.inl:808:13: note: candidate template ignored: target attributes do not match
inline bool eq(const T1& lhs, const T2& rhs) {
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:71:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.inl:21:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/adl/malloc_and_free.h:32:
In file included from /usr/local/cuda-8.0/include/thrust/system/cuda/detail/malloc_and_free.h:23:
In file included from /usr/local/cuda-8.0/include/thrust/system/system_error.h:29:
In file included from /usr/local/cuda-8.0/include/thrust/system/error_code.h:518:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/error_category.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/functional.h:26:
In file included from /usr/local/cuda-8.0/include/thrust/detail/functional/placeholder.h:20:
In file included from /usr/local/cuda-8.0/include/thrust/detail/functional/actor.h:191:
/usr/local/cuda-8.0/include/thrust/detail/functional/actor.inl:41:7: error: host function 'actor' cannot overload host device
function 'actor'
::actor(void)
^
/usr/local/cuda-8.0/include/thrust/detail/functional/actor.h:55:3: note: previous declaration is here
actor(void);
^
In file included from /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp:10:
In file included from /usr/local/cuda-8.0/include/thrust/device_ptr.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/memory.h:25:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.h:173:
In file included from /usr/local/cuda-8.0/include/thrust/detail/reference.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.h:71:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/generic/memory.inl:21:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/adl/malloc_and_free.h:32:
In file included from /usr/local/cuda-8.0/include/thrust/system/cuda/detail/malloc_and_free.h:23:
In file included from /usr/local/cuda-8.0/include/thrust/system/system_error.h:29:
In file included from /usr/local/cuda-8.0/include/thrust/system/error_code.h:518:
In file included from /usr/local/cuda-8.0/include/thrust/system/detail/error_category.inl:22:
In file included from /usr/local/cuda-8.0/include/thrust/functional.h:26:
In file included from /usr/local/cuda-8.0/include/thrust/detail/functional/placeholder.h:20:
In file included from /usr/local/cuda-8.0/include/thrust/detail/functional/actor.h:191:
/usr/local/cuda-8.0/include/thrust/detail/functional/actor.inl:47:7: error: host function 'actor' cannot overload host device
function 'actor'
::actor(const Eval &base)
^
/usr/local/cuda-8.0/include/thrust/detail/functional/actor.h:58:3: note: previous declaration is here
actor(const Eval &base);
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
Error while processing /home-nis/tsungyeh/DeepBench/code/nvidia/rnn_bench.cu.hipify-tmp.

[HIPIFY] info: file 'rnn_bench.cu' statistics:
CONVERTED refs count: 13
UNCONVERTED refs count: 0
CONVERSION %: 100
REPLACED bytes: 235
TOTAL bytes: 17858
CHANGED lines of code: 11
TOTAL lines of code: 433
CODE CHANGED (in bytes) %: 1
CODE CHANGED (in lines) %: 3
TIME ELAPSED s: 0.95
[HIPIFY] info: CONVERTED refs by type:
device: 4
mem: 1
math_func: 3
device_func: 1
include_cuda_main_header: 2
type: 1
numeric_literal: 1
[HIPIFY] info: CONVERTED refs by API:
CUDA Driver API: 1
CUDA RT API: 5
CUBLAS API: 0
CURAND API: 7
[HIPIFY] info: CONVERTED refs by names:
CURAND_RNG_PSEUDO_DEFAULT: 1
cuda.h: 1
cudaDeviceSynchronize: 4
cudaFree: 1
curand: 1
curand.h: 1
curandCreateGenerator: 1
curandDestroyGenerator: 1
curandGenerator_t: 1
curandSetPseudoRandomGeneratorSeed: 1
[HIPIFY] info: UNCONVERTED refs by type:
[HIPIFY] info: UNCONVERTED refs by API:
CUDA Driver API: 0
CUDA RT API: 0
CUBLAS API: 0
CURAND API: 0
[HIPIFY] info: UNCONVERTED refs by names:

[emankov]

The same works for me (Ubuntu 16.04, clang 4.0.0, CUDA 8.0):

$ ./hipify-clang -print-stats interac.cu -- -x cuda --cuda-path=/usr/local/cuda-8.0 --cuda-gpu-arch=sm_50

[HIPIFY] info: file 'interac.cu' statistics:
  CONVERTED refs count: 21
  UNCONVERTED refs count: 0
  CONVERSION %: 100
  REPLACED bytes: 470
  TOTAL bytes: 9444
  CHANGED lines of code: 19
  TOTAL lines of code: 282
  CODE CHANGED (in bytes) %: 5
  CODE CHANGED (in lines) %: 7
  TIME ELAPSED s: 0.19
[HIPIFY] info: CONVERTED refs by type:
  mem: 10
  kern: 2
  event: 7
  thread: 1
  type: 1
[HIPIFY] info: CONVERTED refs by API:
  CUDA Driver API: 0
  CUDA RT API: 21
  CUBLAS API: 0
  CURAND API: 0
[HIPIFY] info: CONVERTED refs by names:
  cudaEventCreate: 2
  cudaEventDestroy: 2
  cudaEventElapsedTime: 1
  cudaEventRecord: 2
  cudaEvent_t: 1
  cudaFree: 2
  cudaLaunchKernel: 2
  cudaMalloc: 2
  cudaMemcpy: 3
  cudaMemcpyDeviceToHost: 1
  cudaMemcpyHostToDevice: 2
  cudaThreadSynchronize: 1
[HIPIFY] info: UNCONVERTED refs by type:
[HIPIFY] info: UNCONVERTED refs by API:
  CUDA Driver API: 0
  CUDA RT API: 0
  CUBLAS API: 0
  CURAND API: 0
[HIPIFY] info: UNCONVERTED refs by names:

But it looks like you ran another source:

[HIPIFY] info: file 'rnn_bench.cu' statistics:

There were clang issues with some thrust versions, but I'd appreciate your sending me (via e-mail) the sources you have really run.

[mattsinc]

Just to be clear, @emankov, this is a different benchmark than Interac. DeepBench is available on GitHub, my guess is that @b8875 was using this benchmark: https://github.com/baidu-research/DeepBench/blob/master/code/nvidia/rnn_bench.cu

(@b8875, please confirm)

Matt

[emankov]

Well, command line contained interac.cu, but the log was taken from rnn_bench.cu. I'll have a look into it as well. thanks.

[mattsinc]

I didn't catch that, good call. @b8875, is your command line a typo?

Matt

[b8875]

Yes, my command line is a typo. The correct command is rnn_bench.cu which is from DeepBench. https://github.com/baidu-research/DeepBench/tree/master/code/nvidia/rnn_bench.cu
I am using master branch of DeepBench git repo.

Tsung Tai

[emankov]

Hi Tsung Tai and Matt,

1. Thrust shipped with CUDA 8.0 has a plenty of issues with clang. We did a substitution of thrust from https://github.com/thrust/thrust, but I do not recommend it for you.
   Clang 6.0 release has just arrived and it supports CUDA 9.0 (not 9.1) and its thrust is more or less supported by clang now. I've just built llvm with clang 6.0 and built hipify-clang with it as well. I did it on Windows, and it works. On Linux I believe it should work all the more.

I got cuDNN from https://developer.nvidia.com/rdp/cudnn-download, DeepBench - from https://github.com/baidu-research/DeepBench.

Here is a command line:
hipify-clang -print-stats k:\GIT\DeepBench\code\nvidia\rnn_bench.cu -- -x cuda --cuda-path="c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0" -Ik:\CUDNN\cudnn-9.0-windows7-x64-v7.1\include -Ik:\GIT\DeepBench\code\kernels -std=c++14 2>&1 | tee log

Similar command line should work on Linux.

2. hipify-clang doesn't support cuDNN yet, but there is a port to hip: https://github.com/ROCmSoftwarePlatform/hipDNN
   So the question is: do you need cuDNN support in hipify-clang?
   Btw, for now cuBlas and cuRand are the only supported libs.

[emankov]

Hi @b8875 and @mattsinc,

Initial cuDNN support is added to hipify-clang. To test it please refer to the updated Readme.md.

[emankov]

1. Thrust support question is answered.
2. Latest cuDNN 1.7.3 is supported at the same level as hipDNN.