Skip to content

v1.3.0 β€” Local LLM & Offline AI

Latest

Choose a tag to compare

@ShiyangZheng ShiyangZheng released this 03 Jun 15:52
· 2 commits to main since this release

What's New

Local LLM Support

  • Download and run Qwen 2.5-7B (or any GGUF model) entirely offline
  • Zero-Dependency Download: Pure Python stdlib + shell/curl fallback β€” no pip install needed
  • Smart GGUF selection: auto-picks best single-file quantization (q3_k_m > q2_k > ...)
  • Built-in health monitoring, warmup progress, and crash recovery

Code-Enforced Group Chat Routing

  • Group chat routing now deterministic (code decides who speaks to whom)
  • LLM only generates content β€” identical conversation flow across cloud and local models
  • User speaks β†’ pets discuss (2-3 rounds) β†’ wrap up to user

Improvements

  • pet-llm-server.py: Shell/Python polyglot β€” works with /bin/sh even without Python
  • pet-group-chat.py: Local LLM support with short prompt optimized for 7B models
  • pet-auto-reply.py: Auto-detect local vs cloud provider
  • Settings UI: New Local LLM tab with model download, server start/stop, health status

Technical

  • Supports macOS 12+ (macOS 13+ ships with built-in Python 3)
  • Bundled llama-server for Apple Silicon acceleration (Metal GPU)
  • ~3.5GB default model download (Qwen 2.5-7B GGUF q3_k_m)