Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA back-end fails with OpenCL header problems #2617

Closed
jeffhammond opened this issue Oct 8, 2020 · 20 comments
Closed

CUDA back-end fails with OpenCL header problems #2617

jeffhammond opened this issue Oct 8, 2020 · 20 comments
Labels
cuda CUDA back-end

Comments

@jeffhammond
Copy link
Contributor

I am using 69825ee:

$ git pull
Already up-to-date.
$ git branch
* sycl
$ git log -n1
commit 69825ee6cee5f0e2827ddd4b5b50fd0a5ba01cdd
Author: vladimirlaz <vladimir.lazarev@intel.com>
Date:   Thu Oct 8 21:56:39 2020 +0300
...

I build like this:

$ git clean -dfx
$ /swtools/python/python-3.5.1/bin/python3 ./buildbot/configure.py --cuda  && /swtools/python/python-3.5.1/bin/python3 ./buildbot/compile.py 
args:Namespace(arm=False, base_branch=None, branch=None, build_number=None, build_type='Release', builder_dir=None, cmake_gen='Ninja', cmake_opt=None, cuda=True, docs=False, l0_headers=None, l0_loader=None, no_assertions=False, no_werror=False, obj_dir=None, pr_number=None, shared_libs=False, src_dir=None, system_ocl=False)
[Cmake Command]: cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=X86;NVPTX -DLLVM_EXTERNAL_PROJECTS=sycl;llvm-spirv;opencl-aot;xpti;libdevice -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=~/ISYCL/llvm/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=~/ISYCL/llvm/llvm-spirv -DLLVM_EXTERNAL_XPTI_SOURCE_DIR=~/ISYCL/llvm/xpti -DLLVM_EXTERNAL_LIBDEVICE_SOURCE_DIR=~/ISYCL/llvm/libdevice -DLLVM_ENABLE_PROJECTS=clang;llvm-spirv;sycl;opencl-aot;xpti;libdevice;libclc -DLIBCLC_TARGETS_TO_BUILD=nvptx64--;nvptx64--nvidiacl -DSYCL_BUILD_PI_CUDA=ON -DLLVM_BUILD_TOOLS=ON -DSYCL_ENABLE_WERROR=ON -DCMAKE_INSTALL_PREFIX=~/ISYCL/llvm/build/install -DSYCL_INCLUDE_TESTS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DLLVM_ENABLE_SPHINX=OFF -DBUILD_SHARED_LIBS=OFF -DSYCL_ENABLE_XPTI_TRACING=ON ~/ISYCL/llvm/llvm
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /swtools/gcc/7.4.0/bin/gcc
-- Check for working C compiler: /swtools/gcc/7.4.0/bin/gcc
-- Check for working C compiler: /swtools/gcc/7.4.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /swtools/gcc/7.4.0/bin/g++
-- Check for working CXX compiler: /swtools/gcc/7.4.0/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
...
LLVM libdir: ~/ISYCL/llvm/build/./lib
LLVM bindir: ~/ISYCL/llvm/build/./bin
LLVM cxx flags: -fno-rtti;-fno-exceptions

clang: ~/ISYCL/llvm/build/./bin/clang
llvm-as: ~/ISYCL/llvm/build/./bin/llvm-as
llvm-link: ~/ISYCL/llvm/build/./bin/llvm-link
opt: ~/ISYCL/llvm/build/./bin/opt
llvm-spirv: LLVM_SPIRV-NOTFOUND

BUILDING nvptx64--
    DEVICE: none (  )
BUILDING nvptx64--nvidiacl
    DEVICE: none (  )
 Testing : nvptx64--nvidiacl
-- OpenCL_INCLUDE_DIRS is missing. Will try to download OpenCL headers from github.com
-- OpenCL_LIBRARIES is missing. Will try to download OpenCL ICD Loader from github.com
-- Including the PI API CUDA backend.
...

The tool fails like this:

/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice saxpy.cc -o saxpy.x
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
In file included from saxpy.cc:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:11:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/cl_ext_intel.h:431:9: error: unknown type name 'cl_properties'
typedef cl_properties cl_mem_properties_intel;
        ^
In file included from saxpy.cc:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:121:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/exception.hpp:15:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:228:7: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES'
      CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES,
      ^
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:229:45: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_HOST_PROPERTIES'
  PI_DEVICE_INFO_QUEUE_ON_HOST_PROPERTIES = CL_DEVICE_QUEUE_ON_HOST_PROPERTIES,
                                            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 errors generated.
make: *** [saxpy.x] Error 1

This is the source code:

#include <iostream>

#include <cmath>
#include <cfloat>

#include "CL/sycl.hpp"

namespace sycl = cl::sycl;

const float xval(1);
const float yval(2);
const float zval(2);
const float aval(3);

const float correct = (zval + aval * xval + yval);

int main(int argc, char * argv[])
{
    if (argc < 2) {
        std::cerr << "Usage: saxpy.x <vector length>" << std::endl;
        return argc;
    }

    size_t length = std::atoi(argv[1]);
    std::cout << "SAXPY with " << length << " elements" << std::endl;

    // host data
    std::vector<float> h_X(length,xval);
    std::vector<float> h_Y(length,yval);
    std::vector<float> h_Z(length,zval);

    sycl::queue q(sycl::default_selector{});

    try {

        const float A(aval);

        sycl::buffer<float,1> d_X { h_X.data(), sycl::range<1>(h_X.size()) };
        sycl::buffer<float,1> d_Y { h_Y.data(), sycl::range<1>(h_Y.size()) };
        sycl::buffer<float,1> d_Z { h_Z.data(), sycl::range<1>(h_Z.size()) };

        q.submit([&](sycl::handler& h) {

#if 0
            auto X = d_X.get_access<sycl::access::mode::read>(h);
            auto Y = d_Y.get_access<sycl::access::mode::read>(h);
            auto Z = d_Z.get_access<sycl::access::mode::read_write>(h);
#else
            sycl::accessor X(d_X,h,sycl::read_only);
            sycl::accessor Y(d_Y,h,sycl::read_only);
            sycl::accessor Z(d_Z,h,sycl::read_write);
#endif

            h.parallel_for<class axpy>( sycl::range<1>{length}, [=] (sycl::id<1> i) {
                Z[i] += A * X[i] + Y[i];
            });
        });
        q.wait();
    }
    catch (sycl::exception & e) {
        std::cout << e.what() << std::endl;
        return 1;
    }

    // check for correctness
    size_t errors(0);
    for (size_t i=0; i<length; ++i) {
        if ( std::abs(h_Z[i] - correct) > FLT_MIN) {
            ++errors;
        }
    }
    if (errors) {
        std::cerr << "There were " << errors << " errors!" << std::endl;
        for (size_t i=0; i<length; ++i) {
            std::cout << i << "," << h_Z[i] << "\n";
        }
        return 1;
    }
    std::cout << "Program completed without error." << std::endl;

    return 0;
}
@bader
Copy link
Contributor

bader commented Oct 9, 2020

I can't reproduce this problem. Here is what I do:

export CUDADIR=/usr/local/cuda-10.1
export PATH=$PATH:$CUDADIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDADIR/lib64
git clone https://github.com/intel/llvm -b sycl
cd llvm/
python3 ./buildbot/configure.py --cuda
python3 ./buildbot/compile.py
vim /tmp/test.cpp # add reproducer source
./build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice /tmp/test.cpp -o /tmp/test.o

@bader
Copy link
Contributor

bader commented Oct 11, 2020

@jeffhammond, were you able to compile the reproducer using these instructions?

I can't reproduce this problem. Here is what I do:

export CUDADIR=/usr/local/cuda-10.1
export PATH=$PATH:$CUDADIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDADIR/lib64
git clone https://github.com/intel/llvm -b sycl
cd llvm/
python3 ./buildbot/configure.py --cuda
python3 ./buildbot/compile.py
vim /tmp/test.cpp # add reproducer source
./build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice /tmp/test.cpp -o /tmp/test.o

@bader bader added the cuda CUDA back-end label Oct 14, 2020
@jeffhammond
Copy link
Contributor Author

Clean build from the latest git and it's still broken. I am using the buildbot recipe just like you.

[jrhammon@pcl-tiergarten src]$ /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++  -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice test.cpp 
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
In file included from test.cpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:11:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/cl_ext_intel.h:431:9: error: unknown type name 'cl_properties'
typedef cl_properties cl_mem_properties_intel;
        ^
In file included from test.cpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:121:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/exception.hpp:15:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:228:7: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES'
      CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES,
      ^
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:229:45: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_HOST_PROPERTIES'
  PI_DEVICE_INFO_QUEUE_ON_HOST_PROPERTIES = CL_DEVICE_QUEUE_ON_HOST_PROPERTIES,
                                            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 errors generated.

@jeffhammond
Copy link
Contributor Author

What I don't understand is why there is any reference to cl_ext_intel.h when I build for CUDA. Why does the compiler need to know anything about Intel OpenCL to do a non-OpenCL CUDA build?

@lfmeadow
Copy link

Look at this:
-- OpenCL_INCLUDE_DIRS is missing. Will try to download OpenCL headers from github.com
-- OpenCL_LIBRARIES is missing. Will try to download OpenCL ICD Loader from github.com
I think you are getting some OpenCL 1.x headers. Get some recent ones somewhere and set those variables mentioned. This might work:
https://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/16840/intel_sdk_for_opencl_applications_2020.2.487.tar.gz
It's kind of ugly but you can probably just untar it and unpack the appropriate RPMs.

@bader
Copy link
Contributor

bader commented Oct 16, 2020

Look at this:
-- OpenCL_INCLUDE_DIRS is missing. Will try to download OpenCL headers from github.com
-- OpenCL_LIBRARIES is missing. Will try to download OpenCL ICD Loader from github.com
I think you are getting some OpenCL 1.x headers. Get some recent ones somewhere and set those variables mentioned. This might work:
https://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/16840/intel_sdk_for_opencl_applications_2020.2.487.tar.gz
It's kind of ugly but you can probably just untar it and unpack the appropriate RPMs.

CMake downloads recent version of OpenCL headers and OpenCL ICD loader from Khronos GitHub. These are good for building DPC++ toolchain, but when compiler is invoked, system paths are searched first and if system doesn't have OpenCL headers installed into system other than the version coming with CUDA SDK, there will be conflict. The solution is described in Get Started Guide document - https://intel.github.io/llvm-docs/GetStartedGuide.html#using-the-dpc-toolchain-on-cuda-platforms.

The OpenCL implementation provided by the CUDA SDK is OpenCL 1.2, which is too old to link with the DPC++ runtime and lacks some symbols.

We recommend installing the low level CPU runtime, following the instructions in the next section.

Instead of installing the low level CPU runtime, it is possible to build and install the Khronos ICD loader, which contains all the symbols required.

@bader
Copy link
Contributor

bader commented Oct 16, 2020 •
edited
Loading

What I don't understand is why there is any reference to cl_ext_intel.h when I build for CUDA. Why does the compiler need to know anything about Intel OpenCL to do a non-OpenCL CUDA build?

I think it might be related to this issue: #1265.

@jeffhammond
Copy link
Contributor Author

Yeah, it would be good to have no dependency on OpenCL headers -- especially Intel OpenCL extensions -- in the CUDA back-end.

@lfmeadow
Copy link

lfmeadow commented Oct 16, 2020 via email

@jeffhammond
Copy link
Contributor Author

It's picking up /swtools/cuda/latest/include/CL/cl.h all over the place, which is not surprising given I am building for CUDA.

@lfmeadow
Copy link

lfmeadow commented Oct 16, 2020 via email

@jeffhammond
Copy link
Contributor Author

Okay. But you agree that users should not have deal with this, right? DPC++ should download the OpenCL headers it needs and use those first.

@lfmeadow
Copy link

lfmeadow commented Oct 16, 2020 via email

@jeffhammond
Copy link
Contributor Author

I've rebuilt DPC++ another 5 times and still cannot get it to pick up the right headers. Whether or not this is solvable, I reject any suggestion that this is something reasonable users are supposed to deal with.

I use GCC 10.2 but it still gets the GCC 4.8 headers, presumably because Linux 3.10 and RHEL/CentOS forces this nonsense on this system.

[jrhammon@pcl-tiergarten dpcpp-tutorial]$ /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -v -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice -L/nfs_home/jrhammon/ISYCL/llvm/build/install/lib -Wl,-rpath=/nfs_home/jrhammon/ISYCL/llvm/build/install/lib saxpy.cc 
clang version 12.0.0 (/nfs_home/jrhammon/ISYCL/llvm/clang 8c667b6745180639c313ea67ad3f175a8212e37b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nfs_home/jrhammon/ISYCL/llvm/build/install/bin
Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.8.5
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /data/swtools/cuda/cuda-10.2, version 10.2
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
 "/data/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2020 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name saxpy.cc -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl -mlink-builtin-bitcode /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /data/swtools/cuda/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0 -internal-isystem /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/ipp/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/mkl/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include -I/swtools/intel/compilers_and_libraries_2017.4.196/linux/daal/include -I/swtools/cuda/latest/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/ipp/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/mkl/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/pstl/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include -I/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/daal/include -I/swtools/ninja/latest/include -I. -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward -internal-isystem /usr/local/include -internal-isystem /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -std=c++17 -fdeprecated-macro -fno-dwarf-directory-asm -fdebug-compilation-dir /nfs_home/jrhammon/dpcpp-tutorial -ferror-limit 3 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -fsycl-unnamed-lambda -o /tmp/saxpy-6fb862.bc -x c++ saxpy.cc
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/swtools/ninja/latest/include"
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include"
ignoring duplicate directory "/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include"
ignoring duplicate directory "/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl"
ignoring duplicate directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5"
ignoring duplicate directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux"
ignoring duplicate directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/ipp/include
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/mkl/include
 /swtools/intel/compilers_and_libraries_2017.4.196/linux/daal/include
 /swtools/cuda/latest/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/ipp/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/mkl/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/pstl/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include
 /data/swtools/intel/compilers_and_libraries_2019.3.199/linux/daal/include
 .
 /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl
 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5
 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/x86_64-redhat-linux
 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/backward
 /usr/local/include
 /data/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/clang/12.0.0/include
 /usr/include
End of search list.
In file included from saxpy.cc:1:
In file included from ./tutorial.hpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:11:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/cl_ext_intel.h:431:9: error: unknown type name 'cl_properties'
typedef cl_properties cl_mem_properties_intel;
        ^
In file included from saxpy.cc:1:
In file included from ./tutorial.hpp:6:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_accessor.hpp:11:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/ONEAPI/atomic_enums.hpp:12:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/access/access.hpp:10:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/common.hpp:121:
In file included from /nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/exception.hpp:15:
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:228:7: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES'
      CL_DEVICE_QUEUE_ON_DEVICE_PROPERTIES,
      ^
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/../include/sycl/CL/sycl/detail/pi.h:229:45: error: use of undeclared identifier 'CL_DEVICE_QUEUE_ON_HOST_PROPERTIES'
  PI_DEVICE_INFO_QUEUE_ON_HOST_PROPERTIES = CL_DEVICE_QUEUE_ON_HOST_PROPERTIES,
                                            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 errors generated.

@lfmeadow
Copy link

lfmeadow commented Oct 16, 2020 via email

@jeffhammond
Copy link
Contributor Author

It was CPATH, which contained /swtools/cuda/latest/include.

$ echo $CPATH
/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include:/swtools/intel/compilers_and_libraries_2017.4.196/linux/ipp/include:/swtools/intel/compilers_and_libraries_2017.4.196/linux/mkl/include:/swtools/intel/compilers_and_libraries_2017.4.196/linux/tbb/include:/swtools/intel/compilers_and_libraries_2017.4.196/linux/daal/include:/swtools/cuda/latest/include:/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/ipp/include:/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/mkl/include:/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/pstl/include:/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include:/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/tbb/include:/data/swtools/intel/compilers_and_libraries_2019.3.199/linux/daal/include:/swtools/ninja/latest/include:

@jeffhammond
Copy link
Contributor Author

The resulting binary doesn't work, but I'm just done with this machine.

[jrhammon@pcl-skx01 dpcpp-tutorial]$ make -f Makefile.isycl -k
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice --gcc-toolchain=/swtools/gcc/latest -L/nfs_home/jrhammon/ISYCL/llvm/build/install/lib -Wl,-rpath=/nfs_home/jrhammon/ISYCL/llvm/build/install/lib saxpy.cc -o saxpy.x
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice --gcc-toolchain=/swtools/gcc/latest -L/nfs_home/jrhammon/ISYCL/llvm/build/install/lib -Wl,-rpath=/nfs_home/jrhammon/ISYCL/llvm/build/install/lib saxpy-usm.cc -o saxpy-usm.x
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice --gcc-toolchain=/swtools/gcc/latest -L/nfs_home/jrhammon/ISYCL/llvm/build/install/lib -Wl,-rpath=/nfs_home/jrhammon/ISYCL/llvm/build/install/lib saxpy-usm2.cc -o saxpy-usm2.x
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
/nfs_home/jrhammon/ISYCL/llvm/build/install/bin/clang++ -O3 -std=c++17 -ferror-limit=3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice --gcc-toolchain=/swtools/gcc/latest -L/nfs_home/jrhammon/ISYCL/llvm/build/install/lib -Wl,-rpath=/nfs_home/jrhammon/ISYCL/llvm/build/install/lib saxpy-usm3.cc -o saxpy-usm3.x
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
[jrhammon@pcl-skx01 dpcpp-tutorial]$ ./saxpy/x
-bash: ./saxpy/x: No such file or directory
[jrhammon@pcl-skx01 dpcpp-tutorial]$ cuda-gdb ./saxpy.x 1000
NVIDIA (R) CUDA Debugger
10.2 release
Portions Copyright (C) 2007-2019 NVIDIA Corporation
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./saxpy.x...(no debugging symbols found)...done.
Attaching to program: /nfs_home/jrhammon/dpcpp-tutorial/saxpy.x, process 1000
ptrace: No such process.
/nfs_home/jrhammon/dpcpp-tutorial/1000: No such file or directory.
(cuda-gdb) run 1000
Starting program: /nfs_home/jrhammon/dpcpp-tutorial/saxpy.x 1000
warning: File "/swtools/gcc/10.2.0/lib64/libstdc++.so.6.0.28-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /swtools/gcc/10.2.0/lib64/libstdc++.so.6.0.28-gdb.py
line to your configuration file "/nfs_home/jrhammon/.cuda-gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/nfs_home/jrhammon/.cuda-gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
warning: File "/nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5.0.0-0-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
SAXPY with 1000 elements
warning: Corrupted shared library list: 0x2aaaaab014f8 != 0x621c10

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaaab6b16 in _dl_relocate_object () from /lib64/ld-linux-x86-64.so.2
(cuda-gdb) bt
#0  0x00002aaaaaab6b16 in _dl_relocate_object () from /lib64/ld-linux-x86-64.so.2
#1  0x00002aaaaaabf5ac in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#2  0x00002aaaaaaba714 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#3  0x00002aaaaaabeacb in _dl_open () from /lib64/ld-linux-x86-64.so.2
#4  0x00002aaaabe0deeb in dlopen_doit () from /usr/lib64/libdl.so.2
#5  0x00002aaaaaaba714 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#6  0x00002aaaabe0e4ed in _dlerror_run () from /usr/lib64/libdl.so.2
#7  0x00002aaaabe0df81 in dlopen@@GLIBC_2.2.5 () from /usr/lib64/libdl.so.2
#8  0x00002aaaab6f3fef in cl::sycl::detail::pi::initializePlugins(std::vector<cl::sycl::detail::plugin, std::allocator<cl::sycl::detail::plugin> >*) ()
   from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#9  0x00002aaaac224e40 in pthread_once () from /usr/lib64/libpthread.so.0
#10 0x00002aaaab6f0a5d in cl::sycl::detail::pi::initialize() () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#11 0x00002aaaab74fd2a in cl::sycl::detail::platform_impl::get_platforms() () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#12 0x00002aaaab7c7b5a in cl::sycl::platform::get_platforms() () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#13 0x00002aaaab7af503 in cl::sycl::device::get_devices(cl::sycl::info::device_type) () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#14 0x00002aaaab7b794d in cl::sycl::device_selector::select_device() const () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#15 0x0000000000404b6d in cl::sycl::queue::queue(cl::sycl::device_selector const&, cl::sycl::property_list const&) ()
#16 0x00000000004027e4 in main ()
(cuda-gdb) 
#0  0x00002aaaaaab6b16 in _dl_relocate_object () from /lib64/ld-linux-x86-64.so.2
#1  0x00002aaaaaabf5ac in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#2  0x00002aaaaaaba714 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#3  0x00002aaaaaabeacb in _dl_open () from /lib64/ld-linux-x86-64.so.2
#4  0x00002aaaabe0deeb in dlopen_doit () from /usr/lib64/libdl.so.2
#5  0x00002aaaaaaba714 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#6  0x00002aaaabe0e4ed in _dlerror_run () from /usr/lib64/libdl.so.2
#7  0x00002aaaabe0df81 in dlopen@@GLIBC_2.2.5 () from /usr/lib64/libdl.so.2
#8  0x00002aaaab6f3fef in cl::sycl::detail::pi::initializePlugins(std::vector<cl::sycl::detail::plugin, std::allocator<cl::sycl::detail::plugin> >*) ()
   from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#9  0x00002aaaac224e40 in pthread_once () from /usr/lib64/libpthread.so.0
#10 0x00002aaaab6f0a5d in cl::sycl::detail::pi::initialize() () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#11 0x00002aaaab74fd2a in cl::sycl::detail::platform_impl::get_platforms() () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#12 0x00002aaaab7c7b5a in cl::sycl::platform::get_platforms() () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#13 0x00002aaaab7af503 in cl::sycl::device::get_devices(cl::sycl::info::device_type) () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#14 0x00002aaaab7b794d in cl::sycl::device_selector::select_device() const () from /nfs_home/jrhammon/ISYCL/llvm/build/install/lib/libsycl.so.5
#15 0x0000000000404b6d in cl::sycl::queue::queue(cl::sycl::device_selector const&, cl::sycl::property_list const&) ()
#16 0x00000000004027e4 in main ()

@lfmeadow
Copy link

lfmeadow commented Oct 16, 2020 via email

@jeffhammond
Copy link
Contributor Author

In fairness to CPATH, CMake is trash and sometimes the only way to convince it to find things is to use CPATH. But the Parallel Studio stuff was not the problem here, nor do I think adding CUDA include path there should be allowed to break everything. DPC++ should not be looking for OpenCL headers anywhere if CUDA is used, which of course is understood per the issue Alexey linked above.

At least now there is a record of my suffering for future internet readers to learn from.

@vpirogov vpirogov mentioned this issue Nov 12, 2020
Closed
@dzarukin
Copy link

Hi! OneDNN library also encountered with the same issue. It's hard to blame CMake that it can't find proper headers automatically since FindOpenCL.cmake (the latest its version) searches for CL/cl.h or OpenCL/cl.h, which is the regular way OpenCL is distributed. DPC++ compiler decision was to encapsulate CL/cl.h into a sycl subfolder: compiler/include/sycl/CL/cl.h. This subdirectory actually messes things up for CMake.

The work around we applied on our side is creating ~CL symlink in include folder to sycl/CL. This way CMake internals are not needed to be modified.

Probably, the proper way to deal with it is to teach CMake work with OpenCL headers in case of DPC++ compiler. But this requires someone to contribute into CMake...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cuda CUDA back-end
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jeffhammond @bader @lfmeadow @dzarukin