Multiplatform Halide building from cmake...

[Honya2000]

Hello,

I'm trying to setup universal multiplatform Halide builder in my project's cmake, so it should build out of the box.
I need to support Windows, WebAssembly, Android, iOS, Linux, etc
Unfortunately this is not so simple. The problem lies in LLVM which is building differently for different platforms.
But i do not get this, if LLVM is mandatory part of Halide why it cannot download and install it automatically during Halide cmake invoking?
So question is how to integrate Halide to the projects via simple:
add_subdirectory(third_party/Halide)
without manual LLVM setup, vcpkg hassle, etc.

Currently - the only way for me to integrate it - is to build Halide manually for each platform and link the project to precompiled libs. Which is quite bad solution, because libs are quite huge in size. Especially when i need Debug and Release libs.

P.S.
Tried to manually build Halide for Windows:
cmake ....\ -G "Visual Studio 17 2022" -Thost=x64 -A x64 --toolchain C:/Projects/Third/vcpkg/scripts/buildsystems/vcpkg.cmake -DCMAKE_TOOLCHAIN_FILE=C:/Projects/Third/vcpkg/scripts/buildsystems/vcpkg.cmake

And get errors:

-- Running vcpkg install
read_contents("C:\Projects\Third\vcpkg\scripts\vcpkgTools.xml"): no such file or directory
-- Running vcpkg install - failed
CMake Error at C:/Projects/Third/vcpkg/scripts/buildsystems/vcpkg.cmake:984 (message):
vcpkg install failed. See logs for more information:
C:\Projects\Third\ai\Halide_build\VS\vcpkg-manifest-install.log
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.30/Modules/CMakeDetermineSystem.cmake:146 (include)
CMakeLists.txt:2 (project)

Not sure what is going on, but vcpkg\scripts\vcpkgTools.xml does not exists...

vcpkg is the latest. Tried all boostraps i found inside vcpkg, didn't help.

Any ideas ?

[Honya2000]

vcpkg installer is finally working.

But Halide cmake is still failing:
-- Could NOT find LLVM (missing: LLVM_DIR)
CMake Error at C:/Program Files/CMake/share/cmake-3.30/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
Could NOT find Halide_LLVM (missing: LLVM_CONFIG Clang_CONFIG
Halide_LLVM_SHARED_LIBRARY WebAssembly X86) (Required is version range
"21...99")
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.30/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
cmake/FindHalide_LLVM.cmake:109 (find_package_handle_standard_args)
C:/Projects/Third/ai/vcpkg/scripts/buildsystems/vcpkg.cmake:939 (_find_package)
CMakeLists.txt:159 (find_package)

[alexreinking]

Currently - the only way for me to integrate it - is to build Halide manually for each platform and link the project to precompiled libs. Which is quite bad solution, because libs are quite huge in size. Especially when i need Debug and Release libs.

Are you using AOT Halide Generators or using the Halide JIT libraries. If you're going AOT, you should only need to build the Halide libraries once for the host platform that's running the Generators. Halide can cross-compile from any supported platform to any supported platform (plus device platforms). You can even build Windows libraries from Linux if you want.

I need to support Windows, WebAssembly, Android, iOS, Linux, etc

This sounds like an AOT scenario. It will probably be best to factor your Halide generators into their own CMake project.

So question is how to integrate Halide to the projects via simple:

add_subdirectory(third_party/Halide)

without manual LLVM setup, vcpkg hassle, etc.

We do not support using add_subdirectory or FetchContent for Halide. Build it separately and use find_package(Halide).

But i do not get this, if LLVM is mandatory part of Halide why it cannot download and install it automatically during Halide cmake invoking?

Many reasons. For starters, it would add at least two hours to the build if we built from source. Downloading binaries opens a whole other can of worms.

If you're on macOS or Linux, there are good binaries available from popular repositories: we support using the Homebrew llvm+lld packages and the official LLVM APT repository.

We also build our own stripped down LLVM binaries in our CI system. If you want to use them, you can download them with uv:

$ uv init --python 3.12
Initialized project `tmp`
$ uv add halide-llvm --index https://pypi.halide-lang.org
Using CPython 3.12.12
Creating virtual environment at: .venv
Resolved 2 packages in 477ms
Prepared 1 package in 1m 14s
Installed 1 package in 89ms
 + halide-llvm==22.1.7

Then CMake will pick up on the LLVM binaries as long as the virtual environment is active. You can also ask the halide-llvm tool for the LLVM installation root (set -DHalide_LLVM_ROOT=<this value>), for instance:

$ source .venv/bin/activate
(tmp) $ halide-llvm
/Users/areinking/tmp/.venv/lib/python3.12/site-packages/halide_llvm/data

Then you would set Halide_LLVM_ROOT=/Users/areinking/tmp/.venv/lib/python3.12/site-packages/halide_llvm/data as either a CMake cache variable (-D) or an environment variable.

[Honya2000]

I'm using both JIT and AOT scenario. JIT is only used in windows debug builds. All other platforms use AOT.
Due to JIT - i cannot use prebuilt Halide releases. They do not include debug libs.

Yesterday i was able to compile LLVM + Halide from sources and integrated those to my project. it is working fine in both AOT and JIT modes on windows build.

But i have trouble using those libs for WebAssembly AOT.
Cmake integration looks like this:

set(HALIDE_DEBUG_TREE "${PROJECT_ROOT_DIR}/third_party/Halide/Debug")
set(HALIDE_RELEASE_TREE "${PROJECT_ROOT_DIR}/third_party/Halide/Release")

if (EMSCRIPTEN)
set(Halide_DIR "${HALIDE_RELEASE_TREE}/lib/cmake/Halide" CACHE PATH "Halide CMake package dir (used for AOT generators)" FORCE)
set(features_simd -wasm_simd128)
set(features_threads -wasm_threads)
set(params_threads threads=false)
set(params threads=false)
set(HL_TARGET wasm-32-wasmrt)
else ()
set(Halide_DIR "${HALIDE_DEBUG_TREE}/lib/cmake/Halide" CACHE PATH "Halide CMake package dir (used for discovery)" FORCE)
endif ()

find_package(Halide REQUIRED)
set(halide_includes "$<TARGET_PROPERTY:Halide::Runtime,INTERFACE_INCLUDE_DIRECTORIES>")

When i tried to configure cmake build for WebAssembly - i get errors:

CMake Error at mediapipe/CMakeLists.txt:2548 (find_package):
Could not find a configuration file for package "Halide" that is compatible
with requested version "".

The following configuration files were considered but not accepted:
C:/..../third_party/Halide/Release/lib/cmake/Halide/HalideConfig.cmake, version: 22.0.0 (64bit)

-- Configuring incomplete, errors occurred!
ninja: error: rebuilding 'build.ninja': subcommand failed

Not sure why cmake wasn't accepted. There are no additional information about the issue.
Any help will be appreciated.

[Honya2000]

Just googled a little bit, and in most cases errors in cmake like this:
Could not find a configuration file for package "..." that is compatible with requested version ""
mean mismatch in 32 / 64 bit configuraion. But... is this really the case for AOT in cross-platform builds?
Or 64 bit generator cannot produce 32 bit WASM code ?

[Honya2000]

Ok, i was able to compile AOT libs for WASM using custom commands in cmake with precompiled generators. Seems like using add_halide_generator / add_halide_library is very tricky in cross compile configs. I didn't figure out how to do this...

[alexreinking]

In a fixed cross-compilation setting, you should be able to set TARGETS on add_halide_library. I'd be very glad to advise if I could see an attempt.