-
Notifications
You must be signed in to change notification settings - Fork 606
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmake failed - nvJPEG #39
Comments
Hi, |
I'm working on a PR that would make |
THXs for all replies. I am working on a public server and do not have access to usr/local. So I tried the first method: $cmake -DBUILD_LMDB=ON -NVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..
-- root:
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
Could NOT find NVJPEG (missing: NVJPEG_INCLUDE_DIR NVJPEG_LIBRARY)
Call Stack (most recent call first):
/usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindNVJPEG.cmake:32 (find_package_handle_standard_args)
cmake/Dependencies.cmake:13 (find_package)
CMakeLists.txt:47 (include)
-- Configuring incomplete, errors occurred!
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeOutput.log".
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeError.log". Unfortunately, it does not work. It looks not reasonable. Or due to I am not familiar with cmake, wrote the command in a wrong way. |
To pass variable to cmake you need to use -D_VARIABLE_NAME_. So try: |
@JanuszL Brilliant!!!! However, some other errors comes out. It seems lack of CMakelist.txt files in DALI-master/third_party/googletest, DALI-master/third_party/benchmark and DALI-master/third_party/pybind11. And another Unkown error. Following are details: $cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..
-- root: /home/smluo/GithubProject/cuda-linux64-nvjpeg
-- Found NVJPEG: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include
-- Found NVJPEG (include: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include, library: /home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a)
CMake Error at cmake/Dependencies.cmake:59 (add_subdirectory):
The source directory
/home/smluo/GithubProject/DALI-master/third_party/googletest
does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
CMake Error at cmake/Dependencies.cmake:66 (add_subdirectory):
The source directory
/home/smluo/GithubProject/DALI-master/third_party/benchmark
does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
-- Found JPEG_TURBO: /home/smluo/anaconda3/include
-- Found JPEG_TURBO (include: /home/smluo/anaconda3/include, library: /home/smluo/anaconda3/lib/libturbojpeg.so)
-- Found OpenCV 2.4.9.1 (libs: opencv_core;opencv_imgproc;opencv_core;opencv_imgproc;opencv_highgui)
CMake Error at cmake/Dependencies.cmake:103 (add_subdirectory):
The source directory
/home/smluo/GithubProject/DALI-master/third_party/pybind11
does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
-- Found LMDB: /usr/include
-- Found LMDB /usr/include : /usr/lib/x86_64-linux-gnu/liblmdb.so
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Found Protobuf version 2 : /usr/include : /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppc_static.a:libnppi_static.a:libculibos.a:libturbojpeg.a:libopencv_core.a:libopencv_imgproc.a:libopencv_imgcodecs.a:liblmdb.a'
-- Adding dependencies to dali: '/usr/local/cuda-8.0/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a;/usr/local/cuda-8.0/lib64/libnppc_static.a;/usr/local/cuda-8.0/lib64/libnppi_static.a;/usr/local/cuda-8.0/lib64/libculibos.a;/home/smluo/anaconda3/lib/libturbojpeg.so;opencv_core;opencv_imgproc;opencv_core;opencv_imgproc;opencv_highgui;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libprotobuf.so'
-- Adding dependencies to dali_test.bin: 'dali'
-- Adding dependencies to dali_benchmark.bin: 'dali'
CMake Error at dali/python/CMakeLists.txt:18 (pybind11_add_module):
Unknown CMake command "pybind11_add_module".
-- Configuring incomplete, errors occurred!
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeOutput.log".
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeError.log". |
It sounds like you didn't do a recursive clone of the git repo. Try this:
|
@cliffwoolley Great it works!!! $ cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg -DJPEG_TURBO_ROOT_DIR=/home/smluo/anaconda3/envs/python35 ..
-- The CXX compiler identification is GNU 5.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PythonInterp: /home/smluo/anaconda3/envs/python35/bin/python (found version "3.5.5")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Git: /usr/bin/git (found version "2.7.4")
-- git Version: v0.0.0-dirty
-- Version: 0.0.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Found CUDA: /usr/local/cuda-8.0 (found suitable version "8.0", minimum required is "8.0")
-- Found NVJPEG: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include (found suitable version "9.0", minimum required is "9.0")
CMake Warning at cmake/Dependencies.cmake:37 (message):
Using nvJPEG 9.0 together with CUDA 8.0 requires NVIDIA drivers compatible
with CUDA 9.0 or later
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
-- Found OpenCV: /home/smluo/anaconda3/envs/python35/include;/home/smluo/anaconda3/envs/python35/include/opencv (found suitable version "3.4.1", minimum required is "2.0")
-- Found PythonLibs: /home/smluo/anaconda3/envs/python35/lib/libpython3.5m.so
-- pybind11 v2.3.dev0
-- Found LMDB: /usr/include (found suitable version "0.9.17", minimum required is "0.9")
-- Found Protobuf: /home/smluo/anaconda3/envs/python35/lib/libprotobuf.so;-pthread (found suitable version "3.5.1", minimum required is "2.0")
-- Enabling TensorFlow TFRecord file format support
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppc_static.a:libnppi_static.a:libculibos.a:libturbojpeg.a:libopencv_core.a:libopencv_imgproc.a:libopencv_highgui.a:libopencv_imgcodecs.a:liblmdb.a'
-- Adding dependencies to dali: '/usr/local/cuda-8.0/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a;/usr/local/cuda-8.0/lib64/libnppc_static.a;/usr/local/cuda-8.0/lib64/libnppi_static.a;/usr/local/cuda-8.0/lib64/libculibos.a;opencv_core;opencv_imgproc;opencv_imgcodecs;/usr/lib/x86_64-linux-gnu/liblmdb.so;/home/smluo/anaconda3/envs/python35/lib/libprotobuf.so'
-- Adding dependencies to dali_test.bin: 'dali'
-- Adding dependencies to dali_benchmark.bin: 'dali'
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Adding dependencies to backend_impl: 'dali'
-- Configuring done
-- Generating done But I still gets errors after I run [ 93%] Linking CXX executable python/nvidia/dali/test/dali_benchmark.bin
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `nppiJpegDecodeGetScanDeadzoneSize'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_benchmark.bin.dir/build.make:266: recipe for target 'dali/python/nvidia/dali/test/dali_benchmark.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_benchmark.bin] Error 1
CMakeFiles/Makefile2:445: recipe for target 'dali/CMakeFiles/dali_benchmark.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_benchmark.bin.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 93%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/operators/op_schema_test.cc.o
[ 94%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/operators/operator_factory_test.cc.o
[ 95%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/executor/executor_test.cc.o
[ 95%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/pipeline_test.cc.o
[ 96%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/op_graph_test.cc.o
[ 97%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/test/dali_test.cc.o
[ 98%] Linking CXX shared module nvidia/dali/backend_impl.cpython-35m-x86_64-linux-gnu.so
[100%] Linking CXX executable python/nvidia/dali/test/dali_test.bin
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `nppiJpegDecodeGetScanDeadzoneSize'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_test.bin.dir/build.make:551: recipe for target 'dali/python/nvidia/dali/test/dali_test.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_test.bin] Error 1
CMakeFiles/Makefile2:407: recipe for target 'dali/CMakeFiles/dali_test.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_test.bin.dir/all] Error 2
[100%] Built target backend_impl
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2 I only paste the errors. |
That one is a bit more of a mystery. From the CMake output I see you are using CUDA 8, support for which was a recent addition that I haven't personally tested at all; that could be the source of the NPPI symbol error. For TurboJpeg I see that CMake found an install in your Conda environment but wasn't able to tell what version of TurboJpeg (libjpeg-turbo) it was; that likely means it was libjpeg-turbo < 1.5 (whereas 1.5 is the minimum requirement and also the version that starts reporting its own version number in the way we presently expect; I'm working on detecting the older versions better). It seems you could either try passing these two additional flags to CMake: But, backing up... is there a reason you can't use the pre-built wheels? |
Hi,
As compilation itself succeed it means that you have libjpeg-turbo > 1.5 (or hacked header which may be one of the causes why it doesn't work).
And check if libjpeg-turbo is properly linked there. |
It seems that we can do nvJpeg to be independent form CUDA runtime. We will try to make that happen in next nvJpeg release. |
@JanuszL @cliffwoolley THX for your replies. For libjpeg-turbo, I installed it by using conda (this is the only way cuz I don't have root privileges on the remote server). I found its version (>1.5) could not identified for some reasons so I comment the 88th line of dali/cmake/CMakelist.txt. Its version could be identified later(I dont know why), so I undo this operation. For CUDA, I updated it to 9.0 and tried to compile DALI again. (python35) [smluo@Node4 ~/GithubProject/dali/build]
$nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
(python35) [smluo@Node4 ~/GithubProject/dali/build]
$make -j"$(nproc)" install
[ 0%] Built target libdali.map
[ 2%] Built target gtest
[ 3%] Built target DALI_PROTO
[ 4%] Built target CAFFE2_PROTO
[ 10%] Built target CAFFE_PROTO
[ 12%] Built target TF_PROTO
[ 21%] Built target benchmark
[ 79%] Built target dali
[ 80%] Linking CXX executable python/nvidia/dali/test/dali_benchmark.bin
[ 81%] Linking CXX shared module nvidia/dali/backend_impl.cpython-35m-x86_64-linux-gnu.so
[ 81%] Linking CXX executable python/nvidia/dali/test/dali_test.bin
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_benchmark.bin.dir/build.make:215: recipe for target 'dali/python/nvidia/dali/test/dali_benchmark.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_benchmark.bin] Error 1
CMakeFiles/Makefile2:443: recipe for target 'dali/CMakeFiles/dali_benchmark.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_benchmark.bin.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_test.bin.dir/build.make:526: recipe for target 'dali/python/nvidia/dali/test/dali_test.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_test.bin] Error 1
CMakeFiles/Makefile2:405: recipe for target 'dali/CMakeFiles/dali_test.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_test.bin.dir/all] Error 2
[ 82%] Built target backend_impl
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2 Then I followed @cliffwoolley 's suggestion, added BTW, I still didn't get what's wrong with benchmark and test part. |
Looking into your CMakeCache.txt file from #48 I don't see JPEG_TURBO_ROOT_DIR or JpegTurbo_* variables. |
I commented I have pulled the latest source code. Unfortunately, it does not work for my situation. Following information may help: conda install -c conda-forge libjpeg-turbo And now when I type libjpeg-turbo 1.5.90 0 conda-forge |
Try to install pkg-config as well:
If you comment out find_package(JpegTurbo 1.5 REQUIRED), then cmake is not able to properly find headers and so library and populate appropriate variables. |
Nice! It works. It finds my JpegTurbo. |
Good. Closing. |
If third_party/*/CMakeLists.txt files are not found, it probably means that the user forgot to do `git clone --recursive`, as in NVIDIA#39 (comment). We can notice this and provide them a more useful message as to how to fix up the situation. Signed-off-by: Cliff Woolley <jwoolley@nvidia.com>
If third_party/*/CMakeLists.txt files are not found, it probably means that the user forgot to do `git clone --recursive`, as in #39 (comment). We can notice this and provide them a more useful message as to how to fix up the situation. Signed-off-by: Cliff Woolley <jwoolley@nvidia.com>
Here is the thing. After I do the cmake step:
$cmake ..
It gives me the error:
I guess the error is about nvJPEG
I downloaded the pre-release version v0.1 and decompressed it. There is no more other instructions on the official website of NVIDIA (https://developer.nvidia.com/nvjpeg)
THX in advance.
The text was updated successfully, but these errors were encountered: