Show and Tell: Ghostlink — Zero-config LAN clustering for ggml-rpc across mixed hardware

[rwilliamspbg-ops]

I’ve been building a distributed inference fabric called Ghostlink that leverages llama.cpp's ggml-rpc backend to automatically pool heterogeneous hardware across a local network without any Kubernetes or manual YAML configuration.

I originally built this because I wanted to run 30B-class models, but my compute was fragmented across an ASUS Zenbook 14 OLED and an iProda mini desktop. I needed a way to instantly stitch their VRAM and RAM together.

Here is how it works: you run the Ghostlink binary on any machine on your LAN. It auto-detects the local hardware (CUDA, DirectML, ROCm, Vulkan, Metal, or NPU), discovers other peer nodes via UDP broadcast/mDNS, and automatically provisions a ggml-rpc cluster to split the tensor graph across the available resources. It wraps all of this in an OpenAI-compatible API.

Key Technical Details:

Native ggml-rpc Integration: The core distribution mechanism relies entirely on llama.cpp's RPC server for the actual cross-machine tensor splitting.

Rust Control Plane: The cluster management, hardware fingerprinting, and scheduling are written in Rust (using lto="thin" and codegen-units=1 for crossing the FFI boundary cleanly).

Experimental Transport Layer: For the transport and latency research layer, it implements zero-copy SPSC ring buffers and AF_XDP kernel-bypass networking. My microbenchmarks on this transport layer are hitting 93.7M ops/s cross-thread and ~5M ops/s on encode/decode round trips.

Hardware-Aware Placement: It handles the quirks of mixed compute environments, dynamically routing workloads depending on whether a node has a discrete GPU, an integrated GPU (like the Radeon 860M), or just CPU.

Real-World Benchmark:

By clustering an AMD Radeon 860M and a 4-core Linux mini PC over a standard LAN, I’ve been able to successfully split and run Qwen3-Coder-30B-A3B-Q3_K_L (13.58 GB) across the network—a model that hard-OOMs either machine individually. For smaller models like Qwen2.5-1.5B-Q4_K_M, the distributed cluster sustains over 50 tok/s.

The repository is fully open-source (MIT) and includes a self-hosted React GUI for managing the cluster and chatting with the models.

Link: https://github.com/rwilliamspbg-ops/Ghostlink

I’m sharing this here because the llama.cpp community understands the nuances of edge inference and memory bandwidth better than anyone. I’d love to get your feedback on the scheduling logic—specifically, how you would approach dynamically re-balancing ggml-rpc layer offloading when a LAN connection experiences sudden latency spikes or jitter.