Llama runtime is failing with CMake version

[mergennachin]

From https://github.com/pytorch/executorch/tree/main/examples/models/llama

Compile:

cmake -DPYTHON_EXECUTABLE=python \
    -DCMAKE_INSTALL_PREFIX=cmake-out \
    -DEXECUTORCH_ENABLE_LOGGING=1 \
    -DCMAKE_BUILD_TYPE=Release \
    -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
    -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
    -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
    -DEXECUTORCH_BUILD_XNNPACK=ON \
    -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
    -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
    -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
    -Bcmake-out .

cmake --build cmake-out -j16 --target install --config Release

Error:

--   FALLBACK_YAML: /Users/mnachin/executorch/configurations/../kernels/portable/functions.yaml
--   OUTPUT_DIR: /Users/mnachin/executorch/cmake-out/configurations
-- Generating operator lib:
--   LIB_NAME: optimized_native_cpu_ops_lib
--   OPS_SCHEMA_YAML: /Users/mnachin/executorch/cmake-out/configurations/merged.yaml
--   ROOT_OPS:
--   INCLUDE_ALL_OPS:
Command - python;-m;codegen.tools.gen_oplist;--output_path=/Users/mnachin/executorch/cmake-out/configurations/optimized_native_cpu_ops_lib/selected_operators.yaml;--ops_schema_yaml_path="/Users/mnachin/executorch/cmake-out/configurations/merged.yaml"
-- Generating kernel bindings:
--   LIB_NAME: optimized_native_cpu_ops_lib
--   FUNCTIONS_YAML: /Users/mnachin/executorch/cmake-out/configurations/merged.yaml
--   CUSTOM_OPS_YAML:
--   ADD_EXCEPTION_BOUNDARY: TRUE
Generated files /Users/mnachin/executorch/cmake-out/configurations/optimized_native_cpu_ops_lib/RegisterCodegenUnboxedKernelsEverything.cpp;/Users/mnachin/executorch/cmake-out/configurations/optimized_native_cpu_ops_lib/Functions.h;/Users/mnachin/executorch/cmake-out/configurations/optimized_native_cpu_ops_lib/NativeFunctions.h
-- Generating operator lib:
--   LIB_NAME: optimized_native_cpu_ops_lib
--   KERNEL_LIBS: optimized_kernels;optimized_portable_kernels
--   DEPS: executorch
CMake Error at third-party/gflags/CMakeLists.txt:73 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.


-- Configuring incomplete, errors occurred!
make: Makefile: No such file or directory
make: *** No rule to make target `Makefile'.  Stop.

[mergennachin]

cc @lucylq

[mergennachin]

Also

(executorch_test_4_8) mnachin@mnachin-mbp executorch % cmake -DPYTHON_EXECUTABLE=python \
    -DCMAKE_INSTALL_PREFIX=cmake-out \
    -DCMAKE_BUILD_TYPE=Release \
    -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
    -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
    -DEXECUTORCH_BUILD_XNNPACK=ON \
    -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
    -Bcmake-out/examples/models/llama \
    examples/models/llama

Error

cmake --build cmake-out/examples/models/llama -j16 --config Release
CMake Warning (dev) at CMakeLists.txt:82 (find_package):
  Policy CMP0144 is not set: find_package uses upper-case <PACKAGENAME>_ROOT
  variables.  Run "cmake --help-policy CMP0144" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  CMake variable EXECUTORCH_ROOT is set to:

    /Users/mnachin/executorch/examples/models/llama/../../..

  For compatibility, find_package is ignoring the variable, but code in a
  .cmake module might still use it.
This warning is for project developers.  Use -Wno-dev to suppress it.

etdump library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
bundled_program library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
flatccrt library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
coreml_util library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
coreml_inmemoryfs library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
coremldelegate library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
mpsdelegate library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
neuron_backend library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
qnn_executorch_backend library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
extension_runner_util library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
extension_training library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
kleidiai library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
vulkan_backend library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
quantized_ops_aot_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
-- executorch: Using source file list /Users/mnachin/executorch/cmake-out/examples/models/llama/runner/../../../../executorch_srcs.cmake
CMake Error at runner/CMakeLists.txt:68 (add_subdirectory):
  The source directory

    /Users/mnachin/executorch/extension/llm/tokenizers/third-party/abseil-cpp

  does not contain a CMakeLists.txt file.


CMake Error at runner/CMakeLists.txt:72 (add_subdirectory):
  The source directory

    /Users/mnachin/executorch/extension/llm/tokenizers/third-party/re2

  does not contain a CMakeLists.txt file.


--
-- ******** Summary ********
--   CMAKE_BUILD_TYPE              : Release
--   CMAKE_CXX_STANDARD            : 17
--   CMAKE_CXX_COMPILER_ID         : AppleClang
--   CMAKE_TOOLCHAIN_FILE          :
--   BUCK2                         :
--   PYTHON_EXECUTABLE             : python
--   FLATC_EXECUTABLE              :
--   EXECUTORCH_ENABLE_LOGGING              :
--   EXECUTORCH_ENABLE_PROGRAM_VERIFICATION :
--   EXECUTORCH_LOG_LEVEL                   :
--   EXECUTORCH_BUILD_ANDROID_JNI           :
--   EXECUTORCH_BUILD_ARM_BAREMETAL         :
--   EXECUTORCH_BUILD_CADENCE               :
--   EXECUTORCH_BUILD_COREML                :
--   EXECUTORCH_BUILD_CPUINFO               : ON
--   EXECUTORCH_BUILD_DEVTOOLS              :
--   EXECUTORCH_BUILD_EXECUTOR_RUNNER       :
--   EXECUTORCH_BUILD_EXTENSION_DATA_LOADER :
--   EXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR :
--   EXECUTORCH_BUILD_EXTENSION_LLM         :
--   EXECUTORCH_BUILD_EXTENSION_MODULE      :
--   EXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL :
--   EXECUTORCH_BUILD_EXTENSION_TENSOR      :
--   EXECUTORCH_BUILD_EXTENSION_TRAINING    :
--   EXECUTORCH_BUILD_FLATC                 :
--   EXECUTORCH_BUILD_GFLAGS                :
--   EXECUTORCH_BUILD_HOST_TARGETS          :
--   EXECUTORCH_BUILD_KERNELS_CUSTOM        : ON
--   EXECUTORCH_BUILD_KERNELS_CUSTOM_AOT    :
--   EXECUTORCH_BUILD_KERNELS_OPTIMIZED     : ON
--   EXECUTORCH_BUILD_KERNELS_QUANTIZED     : ON
--   EXECUTORCH_BUILD_MPS                   :
--   EXECUTORCH_BUILD_NEURON                :
--   EXECUTORCH_BUILD_PTHREADPOOL           : ON
--   EXECUTORCH_BUILD_PYBIND                :
--   EXECUTORCH_BUILD_QNN                   :
--   EXECUTORCH_BUILD_SIZE_TEST             :
--   EXECUTORCH_BUILD_TESTS                 :
--   EXECUTORCH_BUILD_VULKAN                :
--   EXECUTORCH_BUILD_XNNPACK               : ON
-- Configuring incomplete, errors occurred!
make: Makefile: No such file or directory
make: *** No rule to make target `Makefile'.  Stop.

[lucylq]

Note this is for mac; works well on x86.

These build commands succeed on my local mac: https://github.com/pytorch/executorch/tree/main/examples/models/llama#step-3-run-on-your-computer-to-validate

After:

1. Fresh clone of et
2. Pull release/0.6, git submodule sync && git submodule update —init
3. ./install_executorch.sh —pybind xnnpack
4. Commands in the readme

Cmake version:

(executorch) lfq@lfq-mbp executorch % cmake --version
cmake version 3.29.5

For the missing abseil-cpp and re2 repos, I think we have to do:

git submodule update --init --recursive 

to pull in the additional directories

[lucylq]

Closing this issue as it was resolved with cmake version.