-
-
Notifications
You must be signed in to change notification settings - Fork 36
Lazy lifecycle wrappers for backends without native idle eviction (Phase 1.5) #63
Copy link
Copy link
Open
Labels
backend-drivenBackend-driven discoveryBackend-driven discoveryenhancementNew feature or requestNew feature or requestfeatureNew featureNew featureinfrastructureBuild system, CI, deploymentBuild system, CI, deploymentkilo-auto-fixAuto-generated label by KiloAuto-generated label by Kilokilo-triagedAuto-generated label by KiloAuto-generated label by KilomodelsModel management and inferenceModel management and inferenceschedulerResource schedulerResource scheduler
Description
Metadata
Metadata
Assignees
Labels
backend-drivenBackend-driven discoveryBackend-driven discoveryenhancementNew feature or requestNew feature or requestfeatureNew featureNew featureinfrastructureBuild system, CI, deploymentBuild system, CI, deploymentkilo-auto-fixAuto-generated label by KiloAuto-generated label by Kilokilo-triagedAuto-generated label by KiloAuto-generated label by KilomodelsModel management and inferenceModel management and inferenceschedulerResource schedulerResource scheduler
Some backends load their model at process start and have no built-in idle eviction (
sd-server,llama.cpp llama-serverfor the large fallback path,whisper.cpp server). Each needs a small Python proxy that starts the underlying subprocess on first request and stops it after the per-class TTL. Spec:docs/superpowers/specs/2026-04-11-taos-framework-integration-bridge-design.md§Lazy lifecycle wrappers (Phase 1.5).The
sd-serverCPU/Vulkan fallback was pinning ~1.6 GB on the Orange Pi with zero recent calls; this wrapper is the correct fix without patching upstream binaries.Shape:
LLMProxy/QmdClientlifecycle pattern already in the codebaseAcceptance criteria:
sd-server(CPU/Vulkan SD fallback)llama.cpp llama-server(large model fallback path)whisper.cpp server