Skip to content

Network Optimization

Jump to bottom
youngharold edited this page Feb 17, 2026 · 2 revisions

Network Optimization

Network bandwidth is the primary bottleneck for RPC-based inference. This page covers strategies to minimize its impact.

Bandwidth Requirements

During inference, the coordinator sends tensor data to remote GPUs and receives results back. The amount of data depends on:

  • Model size: More parameters = larger tensors transferred per layer
  • Context length: Longer contexts = more KV cache data
  • Batch size: Larger batches (prompt processing) transfer more data than single-token generation

Rough Estimates for 70B Q4

Operation Data per step At GbE (~110 MB/s) At 2.5GbE (~280 MB/s)
Token generation ~10-30 MB Manageable Fast
Prompt processing (512 tok) ~100-300 MB Bottleneck Manageable

2.5GbE Upgrade

The cheapest meaningful upgrade. USB-C to 2.5GbE adapters cost ~$25 each.

Recommended: Realtek RTL8156 chipset adapters (broadly compatible with Linux and Windows).

Setup:

  1. Plug adapters into both machines
  2. Configure static IPs on the 2.5GbE interfaces (or let DHCP handle it)
  3. Update cluster.yaml worker host IPs if they change
  4. Verify with iperf3: 
    # Server (worker)
iperf3 -s
# Client (coordinator)
iperf3 -c 192.168.86.36
    Expect ~2.35 Gbps (~280 MB/s).

Layer Placement Strategy

Not all layers are equal in network cost:

  • Attention layers: High bandwidth — lots of KV cache data. Keep these on the coordinator (local GPUs).
  • FFN layers: Lower bandwidth per compute step. Better candidates for remote GPUs.

llama.cpp's --tensor-split distributes layers proportionally, which is a reasonable default. For advanced optimization, you'd need to manually assign layer ranges — not yet supported by Tightwad but possible with custom llama-server flags.

Jumbo Frames

If your network supports it, enable jumbo frames (MTU 9000) to reduce per-packet overhead:

# Linux
sudo ip link set eth0 mtu 9000

# Verify
ping -M do -s 8972 192.168.86.36

Both ends and all switches in the path must support jumbo frames.

Dedicated Network

For lowest latency, use a direct Ethernet cable between the two machines (no switch). Assign static IPs on a separate subnet:

Coordinator: 10.0.0.1/24
Worker: 10.0.0.2/24

Update cluster.yaml to use these IPs for RPC traffic while keeping the main network for management/API access.

Future: 10GbE

PCIe 10GbE cards (~$50-80 used) would effectively remove the network bottleneck for 70B models. At ~1.1 GB/s, even prompt processing becomes fast. Worth considering if Tightwad becomes a daily-driver setup.

Clone this wiki locally