Skip to content

Lazy lifecycle wrappers for backends without native idle eviction (Phase 1.5) #63

Description

@jaylfc

Some backends load their model at process start and have no built-in idle eviction (sd-server, llama.cpp llama-server for the large fallback path, whisper.cpp server). Each needs a small Python proxy that starts the underlying subprocess on first request and stops it after the per-class TTL. Spec: docs/superpowers/specs/2026-04-11-taos-framework-integration-bridge-design.md §Lazy lifecycle wrappers (Phase 1.5).

The sd-server CPU/Vulkan fallback was pinning ~1.6 GB on the Orange Pi with zero recent calls; this wrapper is the correct fix without patching upstream binaries.

Shape:

  • Listens on the user-facing port
  • Starts the underlying server subprocess on the first inbound request
  • Forwards all subsequent requests to the subprocess
  • Stops the subprocess after the per-class TTL with no requests; restarts on the next request (pays cold-start once)
  • Follows the LLMProxy / QmdClient lifecycle pattern already in the codebase

Acceptance criteria:

  • Wrapper implemented for sd-server (CPU/Vulkan SD fallback)
  • Wrapper implemented for llama.cpp llama-server (large model fallback path)
  • Wrapper implemented for whisper.cpp server
  • Each wrapper is ~50 LOC and fully isolated; upstream binaries need no changes
  • Wrapper honours the per-class TTL from the scheduler's model registry
  • Cold-start latency is logged so the Activity widget can surface it
  • Existing smoke tests pass through the wrappers

Metadata

Metadata

Assignees

No one assigned

    Labels

    backend-drivenBackend-driven discoveryenhancementNew feature or requestfeatureNew featureinfrastructureBuild system, CI, deploymentkilo-auto-fixAuto-generated label by Kilokilo-triagedAuto-generated label by KilomodelsModel management and inferenceschedulerResource scheduler

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions