rpc : send hash when tensor data is above some fixed threshold

[rgerganov]

ref #10095

[rgerganov]

I added support for caching tensors from GGUF files (via multiple -f args) and specifying local cache dir (-d arg).

The problem with caching GGUF files is that they need to stay in RAM the whole time. I tried to keep a map of hash -> (gguf_path, tensor_name) and load tensors on demand but it turned out to be very slow. You can find this version of the patch here.

The good news is that caching large tensors in a local dir and loading them from there works fine.

I am inclined to drop support for caching GGUFs and leave only the cache dir support. Keeping GGUFs in memory also makes the reported available memory inaccurate with the CPU backend.

[ggerganov]

I am inclined to drop support for caching GGUFs and leave only the cache dir support. Keeping GGUFs in memory also makes the reported available memory inaccurate with the CPU backend.

I'm OK with that - it will make the change even simpler.

[rgerganov]

OK, I have left only the -d arg which specifies the cache dir.

Should we try to reuse fs_get_cache_directory() from common.h and put stuff under $HOME/.cache/llama.cpp/?

[ggerganov]

Should we try to reuse fs_get_cache_directory() from common.h and put stuff under $HOME/.cache/llama.cpp/?

Sounds good. Maybe instead of linking libcommon to rpc-server you can copy paste the function for now.

[rgerganov]

I changed the server to accept a -c, --cache flag which enables the local cache. Currently we use $HOME/.cache/llama.cpp/rpc/. Is using subdir OK? We can also save to $HOME/.cache/llama.cpp/rpc-XXXXXXXXXXXXXXXX and keep the cache flat, I don't have strong preferences

[ggerganov]

I changed the server to accept a -c, --cache flag which enables the local cache.

Yes, that's better - the cache path can be controlled with the env variable.

Is using subdir OK?

Yes.

[ggerganov]

Suggested change

	// NOTE: this is copied from common.cpp to avoid linking with libllama
	// NOTE: this is copied from common.cpp to avoid linking with libcommon

[rgerganov]

ok but I don't understand why putting common in target_link_libraries is pulling libllama as dependency. I have a static libcommon.a built, why is not possible to link against it?

[ggerganov]

libcommon already links to libllama:

llama.cpp/common/CMakeLists.txt

Line 141 in f125b8d

target_link_libraries (${TARGET} PRIVATE ${LLAMA_COMMON_EXTRA_LIBS} PUBLIC llama Threads::Threads)

So anything that links to libcommon will indirectly link to libllama. In theory, we can separate all "common" functionality that does not depend on libllama into a separate standalone common library that would be suitable to link in this case.

Btw, on master, the rpc-server example links to libllama, which is not necessary. You can simply remove this dependency:

diff --git a/examples/rpc/CMakeLists.txt b/examples/rpc/CMakeLists.txt
index ae48fb98d..892db89ea 100644
--- a/examples/rpc/CMakeLists.txt
+++ b/examples/rpc/CMakeLists.txt
@@ -1,2 +1,2 @@
 add_executable(rpc-server rpc-server.cpp)
-target_link_libraries(rpc-server PRIVATE ggml llama)
+target_link_libraries(rpc-server PRIVATE ggml)

[rgerganov]

ok, fixed