How to import the path of compiled protobuf in cmake

[wangguangyuan]

My compiled protobuf has been installed to /home/guangyuan/lib/protobuf_3.5.1, how do I specify the path to my protobuf when compiling dali.

[JanuszL]

Hi,
We rely on FindProtobuf.cmake which is used by find_package.
Internally it relies on find_program, find_library, find_path (full code https://github.com/NVIDIA/DALI/blob/master/cmake/modules/FindProtobuf.cmake). They search patches until the first hit in the following order:

• Search paths specified in cmake-specific cache variables
• Search paths specified in cmake-specific environment variables
• Search the paths specified by the HINTS option
• Search the standard system environment variables - in our case PATH, LIB and INCLUDE
• Search cmake variables defined in the Platform files for the current system
• Search the paths specified by the PATHS option or in the short-hand version of the command

I recommend trying to set env variables: PATH, LIB and INCLUDE to values that point to lib, include and protoc executable.

[JanuszL]

If that doesn't work for you please reopen this issue.

[wangguangyuan]

@JanuszL
I tried to import my own compiled protobuf instead of the system being done, but it was not successful. Because I am on the server, I don't have permission, but I have to use my own compiled protobuf. Is there any good way? Just like NVJPEG, it can be set well with NVJPEG_ROOT_DIR.

cmake -DCMAKE_INSTALL_PREFIX=/home/guangyuan/lib/dali - DNVJPEG_ROOT_DIR=/home/guangyuan/lib/cuda-linux64-nvjpeg -DJPEG_TURBO_ROOT_DIR=/home/guangyuan/lib/libjpeg_turbo_1.5.9 -DProtobuf_INCLUDE_DIRS=/home/guangyuan/lib/protobuf_3.5.1/include -DProtobuf_LIBRARIES=/home/guangyuan/lib/protobuf_3.5.1/lib -DProtobuf_PROTOC_LIBRARIES=/home/guangyuan/lib/protobuf_3.5.1/lib -DProtobuf_PROTOC_EXECUTABLE=/home/guangyuan/lib/protobuf_3.5.1/bin/protoc ..

-- The CXX compiler identification is GNU 4.8.5
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- DALI version: 0.3.0
-- The C compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PythonInterp: /home/guangyuan/anaconda3/envs/tensorflow/bin/python (found version "3.6.6")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Git: /usr/bin/git (found version "1.8.3.1")
-- git Version: v0.2.0-24297324-dirty
-- Version: 0.2.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- compiled but failed to run
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "8.0")
-- Found NVJPEG: /home/guangyuan/lib/cuda-linux64-nvjpeg/include (found suitable version "9.0", minimum required is "9.0")
-- JpegTurbo_INCLUDE_DIR (missing: VERSION_VAR)
-- Found OpenCV: /home/guangyuan/lib/opencv-3.10/include/opencv;/home/guangyuan/lib/opencv-3.10/include (found suitable version "3.1.0", minimum required is "2.0")
-- Found PythonLibs: /home/guangyuan/anaconda3/envs/tensorflow/lib/libpython3.6m.so
-- pybind11 v2.3.dev0
CMake Warning at cmake/modules/FindProtobuf.cmake:433 (message):
Protobuf compiler version 3.5.1 doesn't match library version 2.5.0
Call Stack (most recent call first):
cmake/Dependencies.cmake:132 (find_package)
CMakeLists.txt:43 (include)

-- Found Protobuf: /home/guangyuan/lib/protobuf_3.5.1/lib;-pthread (found suitable version "2.5.0", minimum required is "2.0")
-- TensorFlow TFRecord file format support is not available with Protobuf 2
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppc_static.a:libnppicom_static.a:libnppicc_static.a:libnppig_static.a:libculibos.a:libturbojpeg.a:libopencv_core.a:libopencv_imgproc.a:libopencv_highgui.a:libopencv_imgcodecs.a'
-- Adding dependencies to dali: '/usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib64/librt.so;/home/guangyuan/lib/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a;/usr/local/cuda/lib64/libnppc_static.a;/usr/local/cuda/lib64/libnppicom_static.a;/usr/local/cuda/lib64/libnppicc_static.a;/usr/local/cuda/lib64/libnppig_static.a;/usr/local/cuda/lib64/libculibos.a;/home/guangyuan/lib/libjpeg_turbo_1.5.9/lib64/libturbojpeg.so;opencv_core;opencv_imgproc;opencv_imgcodecs'
-- Adding dependencies to dali_test.bin: 'dali'
-- Adding dependencies to dali_benchmark.bin: 'dali'
-- Adding dependencies to dali_tf: 'dali'
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Adding dependencies to backend_impl: 'dali'
-- Adding dependencies to backend_impl: 'dali_tf'
-- Configuring done
-- Generating done
-- Build files have been written to: /home/guangyuan/dali/build
(tensorflow) [guangyuan@172-16-30-205 build]$ make -j8
[ 1%] Running C++ protocol buffer compiler on proto/dali.proto
Scanning dependencies of target gtest
[ 2%] Running C++ protocol buffer compiler on proto/caffe.proto
[ 2%] Running C++ protocol buffer compiler on proto/caffe2.proto
Scanning dependencies of target benchmark
Scanning dependencies of target libdali.map
make[2]: Circular dali/CMakeFiles/libdali.map <- libdali.map dependency dropped.
[ 2%] Built target libdali.map
Scanning dependencies of target CAFFE_PROTO
Scanning dependencies of target CAFFE2_PROTO
[ 3%] Building CXX object dali/pipeline/operators/reader/parser/CMakeFiles/CAFFE2_PROTO.dir/caffe2.pb.cc.o
Scanning dependencies of target DALI_PROTO
[ 3%] Building CXX object dali/pipeline/operators/reader/parser/CMakeFiles/CAFFE_PROTO.dir/caffe.pb.cc.o
[ 4%] Building CXX object dali/pipeline/CMakeFiles/DALI_PROTO.dir/dali.pb.cc.o
[ 5%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/benchmark.cc.o
[ 6%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/colorprint.cc.o
[ 6%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/commandlineflags.cc.o
[ 7%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o
In file included from /home/guangyuan/dali/build/dali/pipeline/dali.pb.cc:4:0:
/home/guangyuan/dali/build/dali/pipeline/dali.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
/home/guangyuan/dali/build/dali/pipeline/dali.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
/home/guangyuan/dali/build/dali/pipeline/dali.pb.h:14:2: error: #error your headers.
#error your headers.
^
In file included from /home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe2.pb.cc:4:0:
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe2.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe2.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe2.pb.h:14:2: error: #error your headers.
#error your headers.
^
In file included from /home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe.pb.cc:4:0:
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe.pb.h:14:2: error: #error your headers.
#error your headers.
^
[ 7%] Building CXX object third_party/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
In file included from /home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe.pb.cc:4:0:
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe.pb.h:23:35: fatal error: google/protobuf/arena.h: No such file or directory
#include <google/protobuf/arena.h>
^
In file included from /home/guangyuan/dali/build/dali/pipeline/dali.pb.cc:4:0:
/home/guangyuan/dali/build/dali/pipeline/dali.pb.h:23:35: fatal error: google/protobuf/arena.h: No such file or directory
#include <google/protobuf/arena.h>
^
In file included from /home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe2.pb.cc:4:0:
/home/guangyuan/dali/build/dali/pipeline/operators/reader/parser/caffe2.pb.h:23:35: fatal error: google/protobuf/arena.h: No such file or directory
#include <google/protobuf/arena.h>
^
compilation terminated.
compilation terminated.
compilation terminated.
[ 8%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/complexity.cc.o
make[2]: *** [dali/pipeline/operators/reader/parser/CMakeFiles/CAFFE_PROTO.dir/caffe.pb.cc.o] Error 1
make[1]: *** [dali/pipeline/operators/reader/parser/CMakeFiles/CAFFE_PROTO.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 9%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/console_reporter.cc.o
[ 10%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/counter.cc.o
make[2]: *** [dali/pipeline/CMakeFiles/DALI_PROTO.dir/dali.pb.cc.o] Error 1
make[1]: *** [dali/pipeline/CMakeFiles/DALI_PROTO.dir/all] Error 2
[ 10%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/csv_reporter.cc.o
make[2]: *** [dali/pipeline/operators/reader/parser/CMakeFiles/CAFFE2_PROTO.dir/caffe2.pb.cc.o] Error 1
make[1]: *** [dali/pipeline/operators/reader/parser/CMakeFiles/CAFFE2_PROTO.dir/all] Error 2
[ 11%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/json_reporter.cc.o
[ 12%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/reporter.cc.o
[ 13%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/sleep.cc.o
[ 13%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/statistics.cc.o
[ 14%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/string_util.cc.o
[ 15%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/sysinfo.cc.o
[ 16%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o
[ 16%] Linking CXX static library libbenchmark.a
[ 16%] Built target benchmark
[ 17%] Linking CXX static library libgtest.a
[ 17%] Built target gtest
make: *** [all] Error 2

[JanuszL]

Hi,
It is not that easy. For nvJpeg we are using our own custom way of localizing it in the system. For protobuf we are relying on CMake provided scripts.
You can try to google it - what I have found with a quick search is https://stackoverflow.com/questions/50239220/cmake-find-protobuf-compiled-from-source, maybe you can try to investigate this idea and make it work somehow.

[shesung]

try -DCMAKE_PREFIX_PATH=/path/to/protobuf

[zxDeepDiver]

try -DCMAKE_PREFIX_PATH=/path/to/protobuf

Thank you.

[uamglx]

try -DCMAKE_PREFIX_PATH=/path/to/protobuf

Thank you