What
The CLI backend shells out to llama-cli but does not support streaming token callbacks or graceful shutdown of the child process.
Why
Without streaming, clients (like cognitiveosd) cannot receive partial completions. This makes the inference server behave like a batch processor rather than a real-time AI engine as specified in the product specs.
How
- Implement streaming via the existing token callback interface in go processes
- Pipe llama-cli stdout and emit SSE chunks per token
- Handle SIGTERM to kill the child process gracefully
What
The CLI backend shells out to llama-cli but does not support streaming token callbacks or graceful shutdown of the child process.
Why
Without streaming, clients (like cognitiveosd) cannot receive partial completions. This makes the inference server behave like a batch processor rather than a real-time AI engine as specified in the product specs.
How