feat(prefill): operationalize remote MLX workers

[FluffyAIcode]

Summary

• replace cache-only production targeting with capability discovery and co-located worker/cache service on the allens Thunderbolt endpoint
• pass exact sink/window/block/model/auth/network compatibility through the worker LaunchAgent installer
• make the head runtime own the network dashboard and expose live remote job, remote import, reused-token, transfer, fallback, and failure counters
• add a cold unique-prefix acceptance probe that fails unless a real remote worker computes and serves KV
• update the operator runbook and add launch-contract/API/dashboard regression tests

Why

ADR 0017's worker implementation was present, but production still launched ADR 0016's empty cache-only peer. The head never gossiped with a PrefillWorker capability, worker defaults mismatched the head's 2048-token window, and the dashboard could not prove that allens served a request.

Test plan

• pytest -q tests/inference_engine/distributed/test_prefill_worker.py tests/inference_engine/distributed/test_prefill_scheduler.py tests/inference_engine/distributed/test_prefill_orchestrator_e2e.py tests/inference_engine/distributed/test_prefill_cache_runtime_fallback.py — 25 passed
• pytest -q tests/inference_engine/bridge/test_prefill_worker_launchd.py tests/inference_engine/network/test_network_state.py tests/inference_engine/network/test_network_api.py — 9 passed
• exact platform-neutral CI selection — 815 passed, 1 skipped
• bash -n deploy/install_prefill_worker_launchd.sh
• plutil -lint deploy/launchd/ai.kakeya.grpc-runtime-prefill.plist
• post-merge: deploy Gemma worker on allens, then run scripts/verify_remote_prefill_e2e.py

Hardware note

The selected allens host has 16 GB unified memory and the Gemma 4-bit model is 15 GB. The model has been synchronized for the requested real test, but deployment may prove an OOM/swap hardware limit; the acceptance probe will not report success without an actual remote job and imported KV.

Made with Cursor