Distributed inference and LoRA training across towers via reticulum

[joelteply]

Problem

Inference and LoRA fine-tuning are locked to the local machine. Can't use the 5090 tower's GPU from a MacBook at work. Can't distribute training across multiple towers.

Vision

Commands.execute('ai/inference', { model: 'coder-32b' }) transparently routes to whichever tower has the model loaded and VRAM available. Same for genome/train. The user doesn't care WHERE it runs — reticulum handles routing.

Architecture

• Discovery: towers announce capabilities (GPU type, VRAM, loaded models) via reticulum
• Routing: inference requests route to best available tower (lowest latency, has model, has VRAM)
• Training: genome/train can target a remote tower (--tower=5090 or auto-select by VRAM)
• Transport: Tailscale (Tailscale mesh network for multi-tower remote inference #323) provides the mesh network, reticulum provides the command routing
• Streaming: inference tokens stream back to caller in real-time (not batch)

Use cases

1. MacBook at work → 5090 at home for inference (via Tailscale)
2. MacBook starts LoRA training on 5090 (32GB VRAM) while continuing to use local system
3. Multiple towers: 5090 handles training, Mac handles UI, phone handles voice
4. Load balancing: if 5090 is training, route inference to Mac's Metal GPU

Implementation path

1. Tailscale mesh (Tailscale mesh network for multi-tower remote inference #323) — network connectivity
2. Tower capability announcement (GPU, models, VRAM available)
3. Commands.execute() remote routing (already designed in remote command delegation)
4. Inference streaming over WebSocket/gRPC
5. Training job dispatch + progress events across towers
6. Model sync: ensure same GGUF/adapter files available on target tower

[joelteply]

Future distributed model strategies across the Grid:

1. Expert-per-node (MoE natural fit)
Each Grid node holds different MoE experts. Router on coordinator decides which node to dispatch to. Qwen3.5-35B-A3B: 5090 holds code+reasoning experts, 3090 holds general+creative experts, Air holds just the router.

2. Pipeline parallelism
Split layers across nodes. Node A handles layers 0-15, Node B handles 16-31. Activations shuttle between nodes via Tailscale/Reticulum. Works for dense models.

3. Tensor parallelism
Split individual layers' weight matrices across nodes. Most bandwidth-intensive but highest utilization. Needs fast interconnect.

4. Dynamic scheduling
The system discovers available VRAM on each node, assigns model shards based on capacity. Nodes join/leave, model reshards automatically.

Hardware available:

• 5090: 32GB VRAM (BigMama)
• 3090: ~24GB VRAM (Toby)
• 5050 laptop: ~8GB VRAM (Toby)
• MacBook Air: 8-16GB unified (coordinator)
• Combined: ~80GB+ — enough for a 70B model

Strategy 1 (expert-per-node) is the most natural fit with our MoE architecture and Grid transport.