cmake build system not properly caching some variables

[ezyang]

Here is the log output I get when I rerun cmake:

(c2isl) [ezyang@devgpu014.ash3 ~/local/caffe2/build] cmake .. -Wno-dev                                   [56/96195]-- The BLAS backend of choice:Eigen
-- Could NOT find NNPACK (missing:  NNPACK_INCLUDE_DIR NNPACK_LIBRARY) 
-- Will try to build NNPACK from source. If anything fails, follow the NNPACK prerequisite installation steps.
CMake Warning at cmake/External/nnpack.cmake:154 (message):
  NNPACK is chosen to be installed, but confu and ninja that are needed by it
  are not installed.  As a result we won't build with NNPACK.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:53 (include)
  CMakeLists.txt:73 (include)


CMake Warning at cmake/Dependencies.cmake:58 (message):
  Not compiling with NNPACK.  Suppress this warning with -DUSE_NNPACK=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)


-- Could NOT find GFlags (missing:  GFLAGS_INCLUDE_DIR GFLAGS_LIBRARY) 
CMake Warning at cmake/External/gflags.cmake:10 (message):
  gflags is not found.  Caffe2 will build without gflags support but it is
  strongly recommended that you install gflags.
Call Stack (most recent call first):
  cmake/External/glog.cmake:2 (include)
  cmake/Dependencies.cmake:65 (include)
  CMakeLists.txt:73 (include)


-- Found glog    (include: /home/ezyang/local/anaconda2/envs/c2isl/include, library: /home/ezyang/local/anaconda2/e
nvs/c2isl/lib/libglog.so)
-- Found system glog install.
CMake Warning at cmake/Dependencies.cmake:84 (message):
  Not compiling with gflags.  Suppress this warning with -DUSE_GFLAGS=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)


-- Found PythonInterp: /home/ezyang/local/anaconda2/envs/c2isl/bin/python (found version "3.6.1") 
-- Found lmdb    (include: /usr/include, library: /usr/lib64/liblmdb.so)
-- Found LevelDB (include: /usr/include, library: /usr/lib64/libleveldb.so)
-- Found Snappy  (include: /usr/include, library: /usr/lib64/libsnappy.so)
-- Could NOT find RocksDB (missing:  RocksDB_INCLUDE_DIR RocksDB_LIBRARIES) 
CMake Warning at cmake/Dependencies.cmake:139 (message):
  Not compiling with RocksDB.  Suppress this warning with -DUSE_ROCKSDB=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)


CMake Warning at cmake/Dependencies.cmake:182 (message):
  Not compiling with OpenCV.  Suppress this warning with -DUSE_OPENCV=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)


-- Found PythonInterp: /home/ezyang/local/anaconda2/envs/c2isl/bin/python (found suitable version "3.6.1", minimum 
required is "2.7") 
-- NumPy ver. 1.13.0 found (include: /home/ezyang/local/anaconda2/envs/c2isl/lib/python3.6/site-packages/numpy/core
/include)
-- Could NOT find pybind11 (missing:  pybind11_INCLUDE_DIR) 
-- Could NOT find MPI_C (missing:  MPI_C_LIBRARIES MPI_C_INCLUDE_PATH) 
-- Could NOT find MPI_CXX (missing:  MPI_CXX_LIBRARIES MPI_CXX_INCLUDE_PATH) 
CMake Warning at cmake/Dependencies.cmake:258 (message):
  Not compiling with MPI.  Suppress this warning with -DUSE_MPI=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)


-- Adding -fopenmp
-- CUDA detected: 8.0
-- Added CUDA NVCC flags for: sm_35
-- Found libcuda: /usr/lib64/libcuda.so
-- Found libnvrtc: /usr/local/cuda-8.0/lib64/libnvrtc.so
-- Found cuDNN: v6.0.21  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Could NOT find NCCL (missing:  NCCL_INCLUDE_DIR NCCL_LIBRARY) 
-- NCCL: /home/ezyang/local/caffe2/third_party/nccl/build/lib/libnccl_static.a
-- Could NOT find CUB (missing:  CUB_INCLUDE_DIR) 
-- NumPy ver. 1.13.0 found (include: /home/ezyang/local/anaconda2/envs/c2isl/lib/python3.6/site-packages/n[0/96195]/include)
-- Could NOT find pybind11 (missing:  pybind11_INCLUDE_DIR) 
-- Could NOT find MPI_C (missing:  MPI_C_LIBRARIES MPI_C_INCLUDE_PATH) 
-- Could NOT find MPI_CXX (missing:  MPI_CXX_LIBRARIES MPI_CXX_INCLUDE_PATH) 
CMake Warning at cmake/Dependencies.cmake:258 (message):
  Not compiling with MPI.  Suppress this warning with -DUSE_MPI=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)


-- Adding -fopenmp
-- CUDA detected: 8.0
-- Added CUDA NVCC flags for: sm_35
-- Found libcuda: /usr/lib64/libcuda.so
-- Found libnvrtc: /usr/local/cuda-8.0/lib64/libnvrtc.so
-- Found cuDNN: v6.0.21  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Could NOT find NCCL (missing:  NCCL_INCLUDE_DIR NCCL_LIBRARY) 
-- NCCL: /home/ezyang/local/caffe2/third_party/nccl/build/lib/libnccl_static.a
-- Could NOT find CUB (missing:  CUB_INCLUDE_DIR) 
-- Could NOT find Gloo (missing:  Gloo_INCLUDE_DIR Gloo_LIBRARY) 
-- CUDA detected: 8.0
-- Found libcuda: /usr/lib64/libcuda.so
-- Found libnvrtc: /usr/local/cuda-8.0/lib64/libnvrtc.so
-- Does not need to define long separately.
-- This compiler has builtin_cpu_supports feature.
-- Turning off deprecation warning due to glog.
-- GCC 4.8.5: Adding gcc and gcc_s libs to link line
-- Include NCCL operators
-- Excluding image processing operators due to no opencv
-- Excluding video processing operators due to no opencv
-- Excluding mkl operators as we are not using mkl
-- MPI operators skipped due to no MPI support
-- Automatically generating missing __init__.py files.
-- 
-- ******** Summary ********
-- General:
--   Git version           : 
--   System                : Linux
--   C++ compiler          : /home/ezyang/ccache/lib/c++
--   C++ compiler version  : 4.8.5
--   Protobuf compiler     : /bin/protoc
--   CXX flags             :  -fopenmp -Wno-deprecated -std=c++11 -fPIC -Wno-narrowing
--   Build type            : Release
--   Compile definitions   : CAFFE2_USE_EIGEN_FOR_BLAS;CAFFE2_USE_GOOGLE_GLOG;EIGEN_MPL2_ONLY
-- 
--   BUILD_SHARED_LIBS     : ON
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.5
--     Python library      : /usr/lib64/libpython2.7.so
--   BUILD_TEST            : ON
--   USE_CUDA              : ON
--     CUDA version        : 8.0
--   USE_CNMEM             : OFF
--   USE_NERVANA_GPU       : OFF
--   USE_GLOG              : ON
--   USE_GFLAGS            : OFF
--   USE_LMDB              : ON
--     LMDB version        : 0.9.18
--   USE_LEVELDB           : ON
--     LevelDB version     : 1.12
--     Snappy version      : 1.1.0
--   USE_OPENCV            : OFF
--   USE_FFMPEG            : 
--   USE_ZMQ               : OFF
--   USE_ROCKSDB           : OFF
--   USE_MPI               : OFF
--   USE_NCCL              : ON
--   USE_NNPACK            : OFF
--   USE_OPENMP            : ON
--   USE_REDIS             : OFF
--   USE_GLOO              : ON
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ezyang/local/caffe2/build

It looks like caffe2 is reprobing for system libraries when it really shouldn't (e.g., glog, PythonInterp, lmdb.)

[deepali-c]

On rerun, cmake is not searching for these, but only prints the status message.
Eg. https://github.com/caffe2/caffe2/blob/master/cmake/Modules/FindGFlags.cmake#47

[ezyang]

@deepali-c I think the trouble is optional deps. I believe that when they are notfound, by default cmake will search for them again.

[deepali-c]

Yes absolutely correct, if dependency is not found then cmake would search for them again in next run.

[pietern]

I think this is nice to have, but not a deal breaker. If we want/need to revisit this later we can open a new issue. Closing this one.