Skip to content

Hardware Setup

Jump to bottom
youngharold edited this page Feb 17, 2026 · 5 revisions

Hardware Setup

Quick Start: Pre-built Binaries

The fastest way to get started. Download matching rpc-server and llama-server from the same llama.cpp release.

Platform Asset name Key binary
Windows CUDA llama-b8079-bin-win-cuda-cu12.2.0-x64.zip rpc-server.exe, llama-server.exe
macOS ARM llama-b8079-bin-macos-arm64.tar.gz rpc-server, llama-server
Linux ROCm Build from source (see below) rpc-server, llama-server

Critical: All binaries must be from the same release. Version mismatches cause silent failures — the coordinator will hang during model loading with no error message.

Worker Setup

Workers run rpc-server to expose their GPU over the network.

Windows (CUDA)

# Extract the release zip to C:\llama\
# Start rpc-server:
C:\llama\rpc-server.exe --host 0.0.0.0 --port 50052

Firewall: Add both a port rule and a program rule:

New-NetFirewallRule -DisplayName 'llama-rpc-port' -Direction Inbound -Protocol TCP -LocalPort 50052 -Action Allow
New-NetFirewallRule -DisplayName 'llama-rpc-exe' -Direction Inbound -Program 'C:\llama\rpc-server.exe' -Action Allow

Multi-GPU workers: Start one rpc-server per GPU on separate ports:

# GPU 0
CUDA_VISIBLE_DEVICES=0 C:\llama\rpc-server.exe --host 0.0.0.0 --port 50052
# GPU 1
CUDA_VISIBLE_DEVICES=1 C:\llama\rpc-server.exe --host 0.0.0.0 --port 50053

As a Windows service (NSSM):

nssm install tightwad-rpc C:\llama\rpc-server.exe --host 0.0.0.0 --port 50052
nssm start tightwad-rpc

macOS (Metal)

# Extract the release tarball
tar xzf llama-b8079-bin-macos-arm64.tar.gz -C ~/llama-server/

# IMPORTANT: Restrict to Metal GPU only
# Without --device MTL0, macOS exposes Metal + CPU as TWO devices,
# which breaks tensor split calculations on the coordinator
~/llama-server/rpc-server --host 0.0.0.0 --port 50052 --device MTL0

Firewall: macOS application firewall blocks unsigned binaries launched without a GUI prompt (e.g., via SSH). Fix:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add ~/llama-server/rpc-server
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp ~/llama-server/rpc-server

Or go to System Settings → Network → Firewall → Options and add rpc-server.

Apple Silicon VRAM: Use recommendedMaxWorkingSetSize (shown at rpc-server startup, e.g., 11453 MiB for M2 16GB) for the vram_gb value in cluster.yaml, not total system RAM.

Linux (ROCm)

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build \
    -DGGML_HIP=ON \
    -DGGML_RPC=ON \
    -DAMDGPU_TARGETS=gfx1100 \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

# Start rpc-server
./build/bin/rpc-server --host 0.0.0.0 --port 50052

gfx1100 = RX 7900 XTX/XT. Check yours with rocminfo | grep gfx.

Linux (CUDA)

cmake -B build -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)
./build/bin/rpc-server --host 0.0.0.0 --port 50052

Coordinator Setup

The coordinator machine runs llama-server with --rpc flags pointing to all workers.

RAM requirement: The coordinator memory-maps the entire GGUF model file into system RAM before distributing tensor slices to workers. This means the coordinator needs enough system RAM for the full model, regardless of how much gets offloaded to remote GPUs. For a 70B Q4_K_M (~40GB GGUF), budget at least 44GB of system RAM on the coordinator (model + KV cache + compute buffers). There is no way to avoid this with llama.cpp's current RPC architecture.

llama-server -m /path/to/model.gguf \
  --host 0.0.0.0 --port 8090 \
  -ngl 999 \
  --rpc 192.168.1.20:50052,192.168.1.30:50052 \
  --tensor-split 0.34,0.26,0.17,0.23 \
  --flash-attn --jinja \
  --ctx-size 8192 -n 4096

Tensor split order: Coordinator local GPUs first (in CUDA device order), then RPC workers in the order listed in --rpc.

Example: 4-GPU Pool

Desktop (coordinator):  4070 Ti Super (16GB) + 3060 (12GB) = local
XPS (rpc-server):       2070 (8GB) = RPC worker
MacBook (rpc-server):   M2 Metal (11GB) = RPC worker

Total: 47GB → fits Llama 3.3 70B Q4_K_M (~40GB)

--tensor-split 0.34,0.26,0.17,0.23
                ^^^^  ^^^^  ^^^^  ^^^^
               4070  3060  2070   M2

Verification

1. Check RPC Workers

From any machine:

# TCP connectivity test
nc -zv 192.168.1.20 50052
nc -zv 192.168.1.30 50052

# Or use Tightwad CLI
tightwad status

2. Watch Coordinator Logs

Successful startup shows tensor distribution across all devices:

rpc_connect: connected to 192.168.1.20:50052
rpc_connect: connected to 192.168.1.30:50052
load_tensors: buffer CUDA0 = 13456 MiB
load_tensors: buffer CUDA1 = 10234 MiB
load_tensors: buffer RPC[192.168.1.20:50052] = 6789 MiB
load_tensors: buffer RPC[192.168.1.30:50052] = 9123 MiB

If any RPC buffer is missing, that worker wasn't connected.

3. Test a Request

curl http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}'

Coexistence with Existing Services

The rpc-server uses port 50052 by default. This doesn't conflict with:

  • Ollama (11434)
  • Existing llama-server instances (8080, 8081, etc.)

However, GPU memory is shared. If a machine runs both Ollama and rpc-server, the coordinator may fail to allocate tensors. Either stop Ollama first or reduce that GPU's vram_gb in cluster.yaml.

Loading Time

Initial model loading transfers tensor data over WiFi/Ethernet to each worker. For a 40GB model with 40% remote:

  • Gigabit Ethernet: ~2-3 minutes
  • WiFi (802.11ac): ~5-10 minutes
  • 2.5GbE: ~1 minute

This is a one-time cost per startup. Inference traffic is much lighter.

Clone this wiki locally