Misc. bug: CPU backend loader does not respect LD_LIBRARY_PATH

[farshidtz]

Name and Version

7154 (064c90d)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

$ cmake -B build -DGGML_NATIVE=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_CURL=OFF
...
$ cmake --build build --config Release -j $(nproc)
...

# Move all shared objects and their symlinks to lib
$ mkdir build/lib
$ mv build/bin/*.so* build/lib/
$ ls -lGg build/lib
total 11872
lrwxrwxrwx 1      17 Nov 25 12:02 libggml-base.so -> libggml-base.so.0
lrwxrwxrwx 1      21 Nov 25 12:02 libggml-base.so.0 -> libggml-base.so.0.9.4
-rwxrwxr-x 1  745680 Nov 25 12:02 libggml-base.so.0.9.4
-rwxrwxr-x 1  922016 Nov 25 12:02 libggml-cpu-alderlake.so
-rwxrwxr-x 1  926112 Nov 25 12:02 libggml-cpu-haswell.so
-rwxrwxr-x 1 1144824 Nov 25 12:02 libggml-cpu-icelake.so
-rwxrwxr-x 1  849888 Nov 25 12:02 libggml-cpu-sandybridge.so
-rwxrwxr-x 1 1378384 Nov 25 12:02 libggml-cpu-sapphirerapids.so
-rwxrwxr-x 1 1144824 Nov 25 12:02 libggml-cpu-skylakex.so
-rwxrwxr-x 1  683456 Nov 25 12:02 libggml-cpu-sse42.so
-rwxrwxr-x 1  679448 Nov 25 12:02 libggml-cpu-x64.so
lrwxrwxrwx 1      12 Nov 25 12:02 libggml.so -> libggml.so.0
lrwxrwxrwx 1      16 Nov 25 12:02 libggml.so.0 -> libggml.so.0.9.4
-rwxrwxr-x 1   55456 Nov 25 12:02 libggml.so.0.9.4
lrwxrwxrwx 1      13 Nov 25 12:02 libllama.so -> libllama.so.0
lrwxrwxrwx 1      20 Nov 25 12:02 libllama.so.0 -> libllama.so.0.0.7154
-rwxrwxr-x 1 2769144 Nov 25 12:02 libllama.so.0.0.7154
lrwxrwxrwx 1      12 Nov 25 12:03 libmtmd.so -> libmtmd.so.0
lrwxrwxrwx 1      19 Nov 25 12:03 libmtmd.so.0 -> libmtmd.so.0.0.7154
-rwxrwxr-x 1  831240 Nov 25 12:03 libmtmd.so.0.0.7154

# It succeeds to load libmtmd.so, libggml.so, etc, but not libggml-cpu*
$ LD_LIBRARY_PATH=build/lib build/bin/llama-server --model gemma-3-4b-it-q4_0.gguf --mmproj mmproj-model-f16-4B.gguf
main: setting n_parallel = 4 and kv_unified = true (add -kvu to disable this)
build: 7154 (064c90d84) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
system info: n_threads = 8, n_threads_batch = 8, total_threads = 32

system_info: n_threads = 8 (n_threads_batch = 8) / 32 | 

init: using 31 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model 'gemma-3-4b-it-q4_0.gguf'
llama_model_load_from_file_impl: no backends are loaded. hint: use ggml_backend_load() or ggml_backend_load_all() to load a backend before calling this function
common_init_from_params: failed to load model 'gemma-3-4b-it-q4_0.gguf', try reducing --n-gpu-layers if you're running out of VRAM
srv    load_model: failed to load model, 'gemma-3-4b-it-q4_0.gguf'
srv    operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

Problem description & steps to reproduce

llama-server, built with GGML_CPU_ALL_VARIANTS=ON flag, fails to load the CPU backend if they aren't in the same directory as the executable:
llama_model_load_from_file_impl: no backends are loaded. hint: use ggml_backend_load() or ggml_backend_load_all() to load a backend before calling this function.

The usual load_backend: loaded CPU backend from ... log message isn't present at the beginning of the logs. Same with llama-cli.

Setting LD_LIBRARY_PATH to the directory with the shared objects does not help. CPU backends are only loaded if they are in the same directory. This is only an issue with libggml-cpu.*.so files. Moving only those back solves the issue:

$ mv build/lib/libggml-cpu* build/bin
$ ./build/bin/llama-cli --version
./build/bin/llama-cli: error while loading shared libraries: libllama.so.0: cannot open shared object file: No such file or directory
$ LD_LIBRARY_PATH=build/lib ./build/bin/llama-cli --version
load_backend: loaded CPU backend from /home/farshid/projects/llama.cpp/build/bin/libggml-cpu-alderlake.so
version: 7154 (064c90d84)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

First Bad Commit

No response

Relevant log output

[o7si]

Hi @farshidtz , You can try adding -DGGML_BACKEND_DIR during the build process:

llama.cpp/ggml/CMakeLists.txt

Line 80 in 583cb83

set(GGML_BACKEND_DIR "" CACHE PATH "ggml: directory to load dynamic backends from (requires GGML_BACKEND_DL")

This will affect the backend search path:

llama.cpp/ggml/src/ggml-backend-reg.cpp

Lines 521 to 531 in 583cb83

	std::vector<fs::path> search_paths;
	if (user_search_path == nullptr) {
	#ifdef GGML_BACKEND_DIR
	search_paths.push_back(fs::u8path(GGML_BACKEND_DIR));
	#endif
	// default search paths: executable directory, current directory
	search_paths.push_back(get_executable_path());
	search_paths.push_back(fs::current_path());
	} else {
	search_paths.push_back(fs::u8path(user_search_path));
	}

However, controlling the search path through build parameters seems a bit odd. Perhaps adding a new environment variable would be a better approach.

[pwilkin]

But it's already supported as an environment variable:

    // check the environment variable GGML_BACKEND_PATH to load an out-of-tree backend
    const char * backend_path = std::getenv("GGML_BACKEND_PATH");
    if (backend_path) {
        ggml_backend_load(backend_path);
    }

[o7si]

Hi @pwilkin , I have a question about GGML_BACKEND_PATH - it seems like it can only load one specific .so file at a time. Is there a way to load multiple .so files, or am I missing something?