Missing flag for ROCM llama.cpp

[Alicimo]

Describe the bug

The llama.cpp package appears to support ROCM. However, the current build misses a key build flag (DAMDGPU_TARGETS) where one specifies your GPU type. Without this, it runs with a default value. The build succeeds, but at runtime, the application will fail if your GPU is not of the default type.

An example error log from this "bug" is documented within the llama.cpp repo here:
ggml-org/llama.cpp#3320

Steps To Reproduce

Steps to reproduce the behaviour:

1. Install llama.cpp with rocm support:

environment.systemPackages = with pkgs; [
  (llama-cpp.override { rocmSupport = true; openblasSupport=false; })
];

2. Download a suitable model, e.g. model
3. llama-cpp-server -m codellama-7b-instruct.Q4_K_M.gguf -ngl 35

Ran on AMD Radeon RX 6900 XT

Expected behavior

The server should start.

Additional context

Overriding is an easy solution, but I would prefer a solution built into nix-pkgs. e.g.,

((pkgs.llama-cpp.overrideAttrs (finalAttrs: previousAttrs: {
    cmakeFlags = (previousAttrs.cmakeFlags ++ [ "-DAMDGPU_TARGETS=gfx1030" ]);
    })).override { rocmSupport = true; openblasSupport=false; });

Note that "gfx1030" is obtained via rocminfo and could be extracted at runtime when not set by the user, i.e., the user would only specify rocmSupport = true. e.g.,

rocminfo | grep gfx | head -1 | awk '{print $2}'

Notify maintainers

@dit7ya
@elohmeier

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.69, NixOS, 23.11 (Tapir), 23.11.2596.c1be43e8e837`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"home-manager-23.11.tar.gz, nixos-23.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

---

Add a 👍 reaction to issues you find important.

[Alicimo]

This is fixed in the unstable branch, so closing, Hopefully the ticket is useful for anyone else using 23.11.