Add all dependencies in the docs

[josemonsalve2]

Hi,

First, I would like to thank you for the most recent additions of compression algorithms.

I was finally able to compile the repo again, but it took me a while since my system did not have the LZ4 lib or the Zlib. Also, for some reason the CMake flag NVCOMP_EXTS_ROOT was not enough.

I build the dependencies by hand and added the flags ZLIB_ROOT, GDEFLATE_ROOT, LZ4_INCLUDE_DIR, and LZ4_LIBRARY to the cmake line, pointing to the appropriate folders.

While I am not sure all this is necessary, it would be nice to have some info about the dependencies.

Thanks!

Jose

[nv-dlasalle]

@josemonsalve2 Can you post the errors you encountered with cmake when trying to build without lz4 and zlib?

[josemonsalve2]

Sure thing.

Here are some outputs:

> cmake ..
-- The C compiler identification is GNU 7.5.0
-- The CUDA compiler identification is NVIDIA 11.0.194
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /hidden/location/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /hidden/location/cuda/cuda-11.0.2/include (found version "11.0.194")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
Release build.
-- Finding CUB
CMake Warning at CMakeLists.txt:92 (find_package):
  By not providing "Findbitcomp.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "bitcomp", but
  CMake did not find one.

  Could not find a package configuration file provided by "bitcomp" with any
  of the following names:

    bitcompConfig.cmake
    bitcomp-config.cmake

  Add the installation prefix of "bitcomp" to CMAKE_PREFIX_PATH or set
  "bitcomp_DIR" to a directory containing one of the above files.  If
  "bitcomp" provides a separate development package or SDK, be sure it has
  been installed.


-- Could NOT find gdeflate (missing: GDEFLATE_INCLUDE_DIR GDEFLATE_LIBRARY)
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
CMake Warning at examples/CMakeLists.txt:20 (message):
  Cannot build gdeflate CPU example without gdeflate and zlib libraries.


CMake Warning at examples/CMakeLists.txt:39 (message):
  Skipping building LZ4 CPU example, as no LZ4 library was found.


-- Configuring done
-- Generating done
-- Build files have been written to: /hidden/location/nvcomp/build2

Then we fix that one with -DNVCOMP_EXTS_ROOT= as mentioned in the instructions. But LZ4 library is not found

cmake .. -DNVCOMP_EXTS_ROOT=`pwd`/../ubuntu20.04/11.0/
-- The C compiler identification is GNU 7.5.0
-- The CUDA compiler identification is NVIDIA 11.0.194
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /hidden/location/cuda/cuda-11.0.2/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /hidden/location/cuda/cuda-11.0.2/include (found version "11.0.194")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
Release build.
-- Finding CUB
-- Found bitcomp
-- Found gdeflate: /hidden/location/nvcomp/ubuntu20.04/11.0/include
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
CMake Warning at examples/CMakeLists.txt:39 (message):
  Skipping building LZ4 CPU example, as no LZ4 library was found.


-- Configuring done
-- Generating done
-- Build files have been written to: /hidden/location/p38/nvcomp/build2

To allow building the LZ4 compiler, I had to use -DLZ4_INCLUDE_DIR= and -DLZ4_LIBRARY=

cmake .. -DNVCOMP_EXTS_ROOT=`pwd`/../ubuntu20.04/11.0/ -DLZ4_INCLUDE_DIR=/hidden/location/include -DLZ4_LIBRARY=/hidden/location/lib/liblz4.so
-- The C compiler identification is GNU 7.5.0
-- The CUDA compiler identification is NVIDIA 11.0.194
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /hidden/location/cuda/cuda-11.0.2/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /hidden/location/cuda/cuda-11.0.2/include (found version "11.0.194")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
Release build.
-- Finding CUB
-- Found bitcomp
-- Found gdeflate: /hidden/location/nvcomp/ubuntu20.04/11.0/include
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
-- Configuring done
-- Generating done
-- Build files have been written to: /hidden/location/nvcomp/build2

On anther system I did not have Zlib so I used -DZLIB_ROOT=. But for some reason I cannot reproduce one of my previous errors related to GDEFLATE.

I think that -DGDEFLATE_ROOT was not necessary. But If I don't have LZ4 or Zlib in regular system location I still need to use -DZLIB_ROOT, -DLZ4_INCLUDE_DIR= and -DLZ4_LIBRARY=.

I hope this helps. If you don't think this is important please feel free to close the issue.

[nv-dlasalle]

@josemonsalve2 Thanks for posting the log.

Those warnings for the examples are expected, but are not related to nvcomp itself, as it does not depend on lz4 or zlib. As you have pointed out, we need to do a much better job messaging this in our cmake files, as well as make it clear in the documentation. We will also be making the examples, tests, and benchmarks not be built by default in future releases.

[josemonsalve2]

That sounds great. I appreciate your quick response.

[josemonsalve2]

Hi,

I was able to reproduce the problem with libz in a different system

cmake -DNVCOMP_EXTS_ROOT=../../ubuntu20.04/11.0/ ../
-- The C compiler identification is GNU 9.3.0
-- The CUDA compiler identification is NVIDIA 11.0.221
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda-11.0/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /usr/local/cuda-11.0/include (found version "11.0.221")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
Release build.
-- Finding CUB
-- Found bitcomp
-- Could NOT find gdeflate (missing: GDEFLATE_INCLUDE_DIR GDEFLATE_LIBRARY)
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
CMake Warning at examples/CMakeLists.txt:20 (message):
  Cannot build gdeflate CPU example without gdeflate and zlib libraries.


CMake Warning at examples/CMakeLists.txt:39 (message):
  Skipping building LZ4 CPU example, as no LZ4 library was found.


-- Configuring done
-- Generating done
-- Build files have been written to: /home/josem/postdoc/nvcomp/build

[josemonsalve2]

NVM, the problem was not with libz, it was with gdeflate. But it was because the -DNVCOMP_EXTS_ROOT must be an absolute path.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[CupidLiuTian]

Hello, i has the similar question with you , i get the extension file to my server and add '-DNVCOMP_EXTS_ROOT=' to cmake,the result is that the cmake procedure still warn can not find gdeflate,bitcomp and ans.detail info as follow:
cmake -DNVCOMP_EXTS_ROOT=/home/test/liutian/external -DBUILD_BENCHMARKS=ON -DBUILD_EXAMPLES=ON -DBUILD_TESTS=ON -DCMAKE_INSTALL_PREFIX=/home/test/liutian/libnew ..
Release build.
-- Finding CUB
-- ans_ROOT=/home/test/liutian/external
NVCOMP_EXTS_ROOT Set Success!
CMake Warning at CMakeLists.txt:107 (find_package):
By not providing "Findbitcomp.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "bitcomp", but
CMake did not find one.

Could not find a package configuration file provided by "bitcomp" with any
of the following names:

bitcompConfig.cmake
bitcomp-config.cmake

Add the installation prefix of "bitcomp" to CMAKE_PREFIX_PATH or set
"bitcomp_DIR" to a directory containing one of the above files. If
"bitcomp" provides a separate development package or SDK, be sure it has
been installed.

CMake Warning at CMakeLists.txt:113 (find_package):
By not providing "Findgdeflate.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "gdeflate",
but CMake did not find one.

Could not find a package configuration file provided by "gdeflate" with any
of the following names:

gdeflateConfig.cmake
gdeflate-config.cmake

Add the installation prefix of "gdeflate" to CMAKE_PREFIX_PATH or set
"gdeflate_DIR" to a directory containing one of the above files. If
"gdeflate" provides a separate development package or SDK, be sure it has
been installed.

CMake Warning at CMakeLists.txt:122 (find_package):
By not providing "Findans.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "ans", but
CMake did not find one.

Could not find a package configuration file provided by "ans" with any of
the following names:

ansConfig.cmake
ans-config.cmake

So how do you solve this problem,wait for your reply,thanks!

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.