Skip to content

Hardware Setup

Jump to bottom
youngharold edited this page Feb 17, 2026 · 5 revisions

Hardware Setup

Prerequisites

Worker Setup (CUDA — Windows)

The worker machine runs rpc-server, one instance per GPU.

Build

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DGGML_RPC=ON
cmake --build build --config Release

Both -DGGML_CUDA=ON and -DGGML_RPC=ON are required. CUDA handles local GPU compute, RPC enables the network server.

Run

Start one rpc-server per GPU on separate ports:

# GPU 0 (RTX 4070 Ti Super)
build\bin\rpc-server.exe -p 50052

# GPU 1 (RTX 3060)
build\bin\rpc-server.exe -p 50053

Each rpc-server auto-detects and binds to the next available CUDA device in order.

Install as Windows Services (NSSM)

For persistent operation:

nssm install hydra-rpc-50052 C:\path\to\rpc-server.exe -p 50052
nssm install hydra-rpc-50053 C:\path\to\rpc-server.exe -p 50053
nssm start hydra-rpc-50052
nssm start hydra-rpc-50053

Bootstrap Script

Or use the provided script (from WSL or Git Bash):

./scripts/install-worker.sh ~/llama.cpp

Coordinator Setup (ROCm — Ubuntu)

The coordinator runs llama-server which loads the model and connects to remote workers.

Install ROCm

Follow the official guide. Verify with:

rocminfo | grep "Marketing Name"

Build

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build \
    -DGGML_HIP=ON \
    -DGGML_RPC=ON \
    -DAMDGPU_TARGETS=gfx1100 \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)
sudo cp build/bin/llama-server /usr/local/bin/

gfx1100 is the target for RX 7900 XTX/XT. Check yours with rocminfo.

Bootstrap Script

./scripts/install-coordinator.sh ~/llama.cpp

Verification

Test Worker Connectivity

From the coordinator machine:

# TCP check
nc -zv 192.168.86.36 50052
nc -zv 192.168.86.36 50053

# Or use Hydra
hydra status

Test Cross-Vendor RPC

Small model test before committing to a 70B:

llama-cli -m small-model.gguf -ngl 99 \
    --rpc 192.168.86.36:50052 \
    -p "Hello world"

If tokens generate, cross-vendor RPC is working.

GPU Assignment

rpc-server binds to CUDA devices in order. If you need specific GPU assignment on a multi-GPU worker:

# Force GPU 0
CUDA_VISIBLE_DEVICES=0 rpc-server.exe -p 50052

# Force GPU 1
CUDA_VISIBLE_DEVICES=1 rpc-server.exe -p 50053

Coexistence with Existing Services

The Windows desktop currently runs GLM-4.7-Flash via llama-server on port 8080. The rpc-server instances use different ports (50052/50053) and don't conflict. However, GPU memory is shared — running both simultaneously may require reducing context sizes.

Clone this wiki locally