Smart model routing — use best available LLM across cluster nodes

[jaylfc]

Summary

When the cluster detects a GPU worker (e.g. Fedora with RTX 3060), the memory system should automatically route LLM extraction to the better/faster model. If the worker goes offline, fall back to on-device models seamlessly.

Architecture

• Orange Pi (host/server): runs taOSmd with MiniLM ONNX embeddings + Qwen3-4B on NPU for extraction
• Fedora (worker/node): runs Ollama with Qwen3-4B on RTX 3060 (10x faster) or potentially Gemma 4 (better quality when supported)
• Smart routing: check worker health, route extraction tasks to fastest available, fallback to local

Model Priority Chain

1. Gemma 4 on GPU worker (when RKLLM supports it) — best quality
2. Qwen3-4B on GPU worker — same quality as Pi, 10x faster
3. Qwen3-4B on Pi NPU — benchmark baseline (72% extraction recall)
4. Regex extraction on Pi CPU — instant fallback (39% recall)

Implementation

• Health check endpoint on workers: GET /api/worker/health
• Model capability advertisement: worker reports available models + speed
• Extraction router: picks best available backend per request
• Automatic failover: if GPU worker times out, route to local NPU
• No user configuration needed — discovery is automatic via cluster

Related

• Add Fedora as cluster worker — test GPU-accelerated memory functions #199 (Fedora as cluster worker)
• Convert Gemma 4 to RKLLM for RK3588 NPU + publish on HuggingFace #197 (Gemma 4 conversion — blocked on RKLLM support)
• taOSmd optimisations: semantic compression, memify, recall scoring, intent-aware retrieval #198 (taOSmd optimisations)

[kilo-code-bot]

This issue appears to be a duplicate of #199.

Add Fedora as cluster worker — test GPU-accelerated memory functions (#199)

Similarity score: 90%

This comment was generated by Kilo Auto-Triage.

[jaylfc]

Not a duplicate — #199 is about adding Fedora as a general cluster worker. #200 is specifically about smart LLM model routing for the memory system (auto-detect best available model across nodes, failover to local). Related but distinct scope.

[jaylfc]

Core infrastructure built:

• ResourceManager.best_model_for_task() — checks available models across local (Ollama, rkllama/qmd NPU) and cluster workers. Prefers GPU > NPU > CPU. RAM-aware: won't recommend a model that doesn't fit in available memory.
• ResourceManager.evaluate_migration() — auto-upgrades to better model when GPU worker joins, auto-downgrades when worker disconnects or gets busy, upgrades back after idle timeout.
• WorkerRegistry — workers report capabilities via heartbeat, offline detection at 90s threshold.
• CatalogPipeline.detect_best_tier() — now checks both Ollama and rkllama/qmd for model availability.

Remaining: wire the retrieval orchestrator to use the best available model at query time (currently static). The pipeline already uses dynamic tier detection.

Code: tinyagentos/scheduling/resource_manager.py, tinyagentos/scheduling/worker_heartbeat.py