Catch Gateway — Phase 1

Product direction (agentic workflows, broader API coverage): see VISION.md.

This README describes only what exists in this repo right now: a local-first, OpenAI-compatible chat gateway with semantic cache (Qdrant) and passthrough to your LLM.

What Phase 1 does

Catch sits between your app and an OpenAI-compatible chat API (local Ollama by default; cloud providers work with config).

For each chat completion request:

1. Semantic lookup — Embed the normalized last user message (plain string content only) and search Qdrant for a similar prior request.
2. Hit — Return the cached completion when similarity is above the configured threshold (skip the LLM).
3. Miss — Forward to {LLM_BASE_URL}/chat/completions, then write back to the cache in the background.

There is a live dashboard at /dashboard for demos and debugging (hits, misses, passthrough).

Stack

Layer	Choice
Gateway service	Rust — single binary, async I/O
HTTP / API	Axum — OpenAI-style POST /v1/chat/completions + static dashboard
Runtime	Tokio
Vector store (semantic cache)	Qdrant (HTTP API; often run via Docker)
Default LLM + embeddings	Ollama — chat at /v1, embeddings via Ollama’s embedding API
Upstream calls	reqwest (HTTP client)

Optional: point LLM_BASE_URL at OpenAI (or any compatible host) and set OPENAI_API_KEY when you leave local Ollama.

Quick start (local testing)

Prerequisites: Rust (stable), Docker (for Qdrant), Ollama.

1. Qdrant (terminal 1):

   docker run -p 6333:6333 qdrant/qdrant

2. Ollama models (terminal 2) — chat + embeddings:

   ollama pull llama3.1 && ollama pull nomic-embed-text

3. Gateway (terminal 3):

   cd catch-gateway
   cp .env.example .env
   cargo run

4. Try it: open http://127.0.0.1:3000/dashboard to watch cache lookup → hit or passthrough → write-back.

If something else (for example Node) is already listening on port 3000, http://localhost:3000 in the browser may hit that app instead of Catch. Use 127.0.0.1 as above, change BIND_ADDR in .env, or free port 3000 for Catch.

---

Testing (engineering)

Always-on (no server): prompt / cache-key logic (from catch-gateway/):

cd catch-gateway
cargo test --test prompt_suite

Live HTTP + dashboard: start Ollama first, then cargo run, then either:

From catch-gateway/:

cd catch-gateway
./scripts/go_live_test.sh

From repo root (Catch-Source-Code/):

./scripts/go_live_test.sh

or manually (see catch-gateway/tests/http/README.md):

cd catch-gateway
cargo test --test http_suite -- --ignored --nocapture

Run everything (ignored HTTP tests skipped unless --ignored):

cd catch-gateway
cargo test

Manual flow (miss → hit, latency, troubleshooting): catch-gateway/TESTING.md.

Configure

See catch-gateway/.env.example. Important fields: LLM_BASE_URL, OLLAMA_HOST, QDRANT_URL, EMBEDDING_MODEL, EMBEDDING_DIM, CACHE_MIN_SCORE, BIND_ADDR. For dashboard “savings” metrics, tune CATCH_ASSUMED_LLM_LATENCY_MS, CATCH_ASSUMED_MONEY_SAVED_PER_HIT_USD, and CATCH_ESTIMATED_USD_PER_1K_TOKENS.

Behavior (Phase 1)

• Cache key: minified last user message (content as a plain string only).
• Minify: collapses whitespace; removes please, thank you / thanks, kindly (case-insensitive).
• Upstream: POST {LLM_BASE_URL}/chat/completions after minifying user message contents.
• OpenAI: LLM_BASE_URL=https://api.openai.com/v1 and OPENAI_API_KEY.