Misc. bug: Regression: distribution contains duplicate dylibs (macOS)

[abra-code]

Name and Version

version: 7021 (4a5b8af)
built with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.6.0

Operating systems

Mac

Which llama.cpp modules do you know to be affected?

libllama (core library)

Command line

n/a

Problem description & steps to reproduce

I noticed the pre-built archives now contain duplicate dylibs - one unversioned and one versioned. They look like identical copies. The zips are several megabytes larger now. The executables are linking only to versioned dylibs - and the version is "0" - not great.
See the logs.
I have not checked archives for other platforms.

First Bad Commit

4a5b8af

Relevant log output

ls -la /Users/tkukielk/Downloads/llama-arm64-b7021/bin 
total 83016
drwxr-xr-x  53 tkukielk  staff     1696 Nov 12 18:57 .
drwxr-xr-x   4 tkukielk  staff      128 Nov 12 18:57 ..
-rw-r--r--   1 tkukielk  staff   134022 Nov 11 11:21 ggml-common.h
-rw-r--r--   1 tkukielk  staff    15114 Nov 11 11:21 ggml-metal-impl.h
-rw-r--r--   1 tkukielk  staff   392255 Nov 11 11:21 ggml-metal.metal
-rwxr-xr-x   1 tkukielk  staff   651608 Nov 11 11:21 libggml-base.0.9.4.dylib
-rwxr-xr-x   1 tkukielk  staff   651608 Nov 11 11:21 libggml-base.0.dylib
-rwxr-xr-x   1 tkukielk  staff   651608 Nov 11 11:21 libggml-base.dylib
-rwxr-xr-x   1 tkukielk  staff    58536 Nov 11 11:21 libggml-blas.0.9.4.dylib
-rwxr-xr-x   1 tkukielk  staff    58536 Nov 11 11:21 libggml-blas.0.dylib
-rwxr-xr-x   1 tkukielk  staff    58536 Nov 11 11:21 libggml-blas.dylib
-rwxr-xr-x   1 tkukielk  staff   712008 Nov 11 11:21 libggml-cpu.0.9.4.dylib
-rwxr-xr-x   1 tkukielk  staff   712008 Nov 11 11:21 libggml-cpu.0.dylib
-rwxr-xr-x   1 tkukielk  staff   712008 Nov 11 11:21 libggml-cpu.dylib
-rwxr-xr-x   1 tkukielk  staff   725624 Nov 11 11:21 libggml-metal.0.9.4.dylib
-rwxr-xr-x   1 tkukielk  staff   725624 Nov 11 11:21 libggml-metal.0.dylib
-rwxr-xr-x   1 tkukielk  staff   725624 Nov 11 11:21 libggml-metal.dylib
-rwxr-xr-x   1 tkukielk  staff   129224 Nov 11 11:21 libggml-rpc.0.9.4.dylib
-rwxr-xr-x   1 tkukielk  staff   129224 Nov 11 11:21 libggml-rpc.0.dylib
-rwxr-xr-x   1 tkukielk  staff   129224 Nov 11 11:21 libggml-rpc.dylib
-rwxr-xr-x   1 tkukielk  staff    59520 Nov 11 11:21 libggml.0.9.4.dylib
-rwxr-xr-x   1 tkukielk  staff    59520 Nov 11 11:21 libggml.0.dylib
-rwxr-xr-x   1 tkukielk  staff    59520 Nov 11 11:21 libggml.dylib
-rwxr-xr-x   1 tkukielk  staff  1911816 Nov 11 11:22 libllama.0.0.7021.dylib
-rwxr-xr-x   1 tkukielk  staff  1911816 Nov 11 11:22 libllama.0.dylib
-rwxr-xr-x   1 tkukielk  staff  1911816 Nov 11 11:22 libllama.dylib
-rwxr-xr-x   1 tkukielk  staff   621528 Nov 11 11:22 libmtmd.0.0.7021.dylib
-rwxr-xr-x   1 tkukielk  staff   621528 Nov 11 11:22 libmtmd.0.dylib
-rwxr-xr-x   1 tkukielk  staff   621528 Nov 11 11:22 libmtmd.dylib
-rw-r--r--   1 tkukielk  staff     1078 Nov 11 11:23 LICENSE
-rw-r--r--   1 tkukielk  staff     1053 Nov 11 11:23 LICENSE-curl
-rw-r--r--   1 tkukielk  staff     1074 Nov 11 11:23 LICENSE-httplib
-rw-r--r--   1 tkukielk  staff     1075 Nov 11 11:23 LICENSE-jsonhpp
-rw-r--r--   1 tkukielk  staff     1472 Nov 11 11:23 LICENSE-linenoise
-rwxr-xr-x   1 tkukielk  staff  2384944 Nov 11 11:23 llama-batched-bench
-rwxr-xr-x   1 tkukielk  staff   394456 Nov 11 11:23 llama-bench
-rwxr-xr-x@  1 tkukielk  staff  2405928 Nov 11 11:23 llama-cli
-rwxr-xr-x   1 tkukielk  staff  2407384 Nov 11 11:23 llama-cvector-generator
-rwxr-xr-x   1 tkukielk  staff  2391872 Nov 11 11:23 llama-export-lora
-rwxr-xr-x   1 tkukielk  staff    35472 Nov 11 11:21 llama-gemma3-cli
-rwxr-xr-x   1 tkukielk  staff    76064 Nov 11 11:23 llama-gguf-split
-rwxr-xr-x   1 tkukielk  staff  2451712 Nov 11 11:23 llama-imatrix
-rwxr-xr-x   1 tkukielk  staff    35472 Nov 11 11:21 llama-llava-cli
-rwxr-xr-x   1 tkukielk  staff    35472 Nov 11 11:21 llama-minicpmv-cli
-rwxr-xr-x   1 tkukielk  staff  2388896 Nov 11 11:23 llama-mtmd-cli
-rwxr-xr-x   1 tkukielk  staff  2443568 Nov 11 11:23 llama-perplexity
-rwxr-xr-x   1 tkukielk  staff   287680 Nov 11 11:23 llama-quantize
-rwxr-xr-x   1 tkukielk  staff    35472 Nov 11 11:21 llama-qwen2vl-cli
-rwxr-xr-x   1 tkukielk  staff  1981144 Nov 11 11:23 llama-run
-rwxr-xr-x   1 tkukielk  staff  4634528 Nov 11 11:23 llama-server
-rwxr-xr-x   1 tkukielk  staff   246000 Nov 11 11:23 llama-tokenize
-rwxr-xr-x   1 tkukielk  staff  2473816 Nov 11 11:23 llama-tts
-rwxr-xr-x   1 tkukielk  staff   131848 Nov 11 11:21 rpc-server


Looks like only the versioned dylibs are linked into the executables like llama-server:
 
otool -l /Users/tkukielk/Downloads/llama-arm64-b7021/bin/llama-server | grep '@rpath'
         name @rpath/libmtmd.0.dylib (offset 24)
         name @rpath/libllama.0.dylib (offset 24)
         name @rpath/libggml.0.dylib (offset 24)
         name @rpath/libggml-cpu.0.dylib (offset 24)
         name @rpath/libggml-blas.0.dylib (offset 24)
         name @rpath/libggml-metal.0.dylib (offset 24)
         name @rpath/libggml-rpc.0.dylib (offset 24)
         name @rpath/libggml-base.0.dylib (offset 24)

[abra-code]

Correction: these are not duplicates. They are triplicates.

[taronaeo]

see #17091. its supposed to be a symlink but i guess archiving it makes it an actual copy of the linked library.

[furrysalamander]

Oh yeah, I hate that that happens with zip files.

TIL that you can preserve symlinks in zip files:
https://linux.die.net/man/1/zip#:~:text=%2Dy,and%20by%20links.

And apparently they haven't added an option to the upload-artifact action to support it.
actions/upload-artifact#93

I guess that's another reason why .tar.gz etc is more popular for platforms that support it. But I have the feeling that if we switched to tar we'd break people's workflow.

BTW, I know that it's "annoying" that the linked .so is major version 0, but this is actually a good thing. In the event that llama.cpp needs to make a breaking change at some point in the future, bumping that will indicate to downstream SW that the newer version is incompatible at an interface level, and theoretically would even allow you to have the newer and older versions installed side by side.

I'm open to help work through this if we can figure out what the best course of action is. I guess in the worst case we could just only include the major version symlink in the zips.

[taronaeo]

I guess that's another reason why .tar.gz etc is more popular for platforms that support it. But I have the feeling that if we switched to tar we'd break people's workflow.

It might not be a bad idea to start introducing .tar.gz first then slowly deprecate support for .zip, if this helps with the symlink and sizing issues.