Direct communication between Client and LLM Inference Engine

[dan-homebrew]

Deliverable

To deliver by EOD Tuesday

• Revert to initial architecture (Client and LLM communicate directly, SSEs streamed directly to Client)
• Simple AuthZ for LLM Inference Server

Problem

• See bug: llama 7B on llama.cpp on Apple Silicon is responding at 4 tokens per minute #38
• Latency due to way we handle LLM responses
  • Serverless function "collates" LLM responses before persisting in Hasura
  • Hasura is responsible for updating client on frontend
• Bigger issue identified
  • Strategy for handling LLM persistence w/o introducing latency
  • Strategy for making LLM conversations stateful

Private Zenhub Image

Private Zenhub Image