Skip to content

Installation

Jump to bottom
gramps edited this page Jul 20, 2026 · 1 revision

← Back to Home

Installation & Configuration

Architecture Overview

cAIc splits into two machine roles:

  • Coordinator — runs the FastAPI app, broker, database, and all CPU-bound services. Does not need a GPU.
  • Workers — run only llama-server for GPU inference. No database, no HTTP API, no orchestration overhead.

You can start with a single machine acting as both coordinator and worker, then split off workers as your hardware fleet grows.

Prerequisites

Dependency Minimum Notes
Python 3.12+ 3.13 recommended
OS Linux WSL2 on Windows works. macOS untested but may work with changes.
RAM 8 GB 16 GB+ recommended for coordinator with RAG
Disk 1 GB Plus model files (~4–10 GB each)
GPU Optional Required for usable inference speed

Python packages (coordinator)

fastapi>=0.115.0
uvicorn[standard]>=0.32.0
httpx>=0.27.0
pypdf>=5.0.0
python-multipart>=0.0.9
aio-pika>=9.0.0
psutil>=5.9.0

Install: pip install -r requirements.txt

Optional external services

Service Port Purpose Required?
llama-server 8081 LLM inference (OpenAI-compat) Yes
RabbitMQ 5672 AMQP broker for cluster messaging No (single-node skip)
Qdrant 6333 Vector database for RAG No
SearXNG 8888 Privacy-respecting web search No
Phi-4-mini 8083 Query triage classification No (falls back to keywords)
Ollama 11434 Text embeddings for RAG No (if RAG disabled)

All optional services gracefully degrade when absent.

Quick Start (Docker)

git clone https://github.com/mikeshallop/caic.git && cd caic
scripts/setup.sh           # generates .env, secrets, pulls default model (~4.6GB)
docker compose up -d       # boots cAIc + Qdrant + RabbitMQ + SearXNG + llama-server + Ollama

Point a browser at http://localhost:8080 and you're chatting.

Default Model

The setup wizard downloads Qwen2.5-7B-Instruct (Q4_K_M, ~4.6 GB) by default.

Why this model:

  • Fits in 6 GB VRAM — runs on mid-range GPUs (RX 6600 XT, RTX 3060, etc.)
  • Instruction-tuned — handles chat, code, and reasoning without fine-tuning
  • Q4_K_M quantization — best balance of quality and speed for consumer hardware
  • GGUF format — runs natively in llama.cpp with no conversion step

Swap it for any .gguf model you prefer. cAIc's query-routing works with whatever you put in ./models/.

Quick Start (Bare Metal)

1. Clone the repo

git clone https://github.com/mikeshallop/caic.git
cd caic

2. Install Python dependencies

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install psutil  # for hardware stats

3. Set up llama-server

The default model is qwen2.5-7b-instruct (7B parameters, instruction-tuned). You can use any GGUF model.

# Download llama-server binary
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
chmod +x llama-server

# Place a GGUF model file
mkdir -p models
# Download from HuggingFace, e.g.:
# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf \
#   https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q5_k_m.gguf

# Run llama-server
./llama-server \
  --host 0.0.0.0 --port 8081 \
  --model models/qwen2.5-7b-instruct-q5_k_m.gguf \
  --ctx-size 4096 \
  --embeddings \
  --logprobs \
  --n-gpu-layers 99

Verify it's running: curl http://localhost:8081/health

4. Configure cAIc

Copy and edit configuration via environment variables:

export LLAMA_SERVER_BASE=http://localhost:8081
export CAIC_ADMIN_PIN=1234           # change this!
export CAIC_ALLOW_DEFAULT_PIN=true   # set false after first login

Key environment variables:

Variable Default Purpose
LLAMA_SERVER_BASE http://192.168.50.108:8081 llama-server URL
OLLAMA_BASE http://localhost:11434 Embeddings endpoint
CAIC_ADMIN_PIN (auto-required) 4-digit admin PIN
CAIC_ALLOW_DEFAULT_PIN false Allow weak PIN in dev
CAIC_COMPLETIONS_API_KEY (auto-generated) Bearer token for /v1/chat/completions
CAIC_AMQP_URL (file-based) RabbitMQ connection string
CAIC_ALLOWED_CIDRS (LAN defaults) IP allowlist CIDRs
CAIC_TRUSTED_ORIGINS (none) Additional CORS origins
CAIC_TRUST_X_FORWARDED_FOR false Trust reverse proxy IPs
QDRANT_URL http://192.168.50.108:6333 Qdrant vector DB URL

5. Run cAIc

uvicorn app:app --host 0.0.0.0 --port 8080 --reload

Open http://localhost:8080 in your browser. Click "Admin Login" and enter your PIN.

6. (Optional) Set up as a systemd service

[Unit]
Description=cAIc Cluster AI Chat
After=network.target

[Service]
Type=simple
User=gramps
Group=gramps
WorkingDirectory=/opt/jarvischat
ExecStart=/opt/jarvischat/venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=5
Environment=LLAMA_SERVER_BASE=http://localhost:8081
Environment=CAIC_ADMIN_PIN=1319
Environment=CAIC_ALLOW_DEFAULT_PIN=true
Environment=CAIC_COMPLETIONS_API_KEY=caic-sk-...

[Install]
WantedBy=multi-user.target

Save to /etc/systemd/system/caic.service, then:

sudo systemctl daemon-reload
sudo systemctl enable --now caic.service

Cluster Setup

Coordinator with RabbitMQ

# Install RabbitMQ
apt install rabbitmq-server
systemctl enable --now rabbitmq-server

# Create user and vhost
rabbitmqctl add_user caic "$(openssl rand -hex 20)"
rabbitmqctl add_vhost caic
rabbitmqctl set_permissions -p caic caic ".*" ".*" ".*"

# Save password for cAIc
echo -n "$PASSWORD" > /home/gramps/.caic_amqp_secret
chmod 600 /home/gramps/.caic_amqp_secret

Start cAIc with CAIC_AMQP_URL set (or it reads from the secret file).

Adding a Worker Node

Each worker machine needs only llama-server and the node agent.

# Install llama-server
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
chmod +x llama-server

# Install node agent deps
pip install aio-pika httpx psutil

# Configure
mkdir -p /etc/caic
cat > /etc/caic/node-agent.conf << 'EOF'
[agent]
node_name = $(hostname)
node_ip = $(hostname -I | awk '{print $1}')
node_type = worker
capabilities = llm
amqp_url = amqp://caic:PASSWORD@COORDINATOR_IP:5672/caic
llama_port = 8081
models_dir = /var/lib/caic/models
active_model = qwen2.5-7b-instruct-q5_k_m.gguf
EOF

# Start node agent (from repo checkout)
python3 /opt/caic/node_agent/agent.py

Configuration Reference

Payload & Rate Limits

Tunable in config.py or via environment overrides:

Setting Default Description
SESSION_TIMEOUT_SECONDS 90 Session idle timeout
MAX_PIN_ATTEMPTS 5 PIN lockout threshold
PIN_LOCKOUT_SECONDS 300 PIN lockout duration
RATE_WINDOW_SECONDS 60 Rate limit window
RL_CHAT_PER_WINDOW 24 Max chat requests per window
RL_SEARCH_PER_WINDOW 16 Max search requests per window
BODY_LIMIT_CHAT_BYTES 128 KB Max chat payload
MAX_UPLOAD_BYTES 20 MB Max file upload

RAG Settings

Setting Default Description
RAG_MAX_VECTORS 50000 Max vectors before eviction
RAG_EVICTION_HIGH_WATER 0.80 Trigger eviction at 80%
RAG_EVICTION_LOW_WATER 0.20 Stop eviction at 20%
RAG_PINNED_SOURCES upload, profile Never evict these sources
RAG_GRACE_HOURS 1 Min age before eviction eligible

Model Configuration

DEFAULT_MODEL in config.py (default: "qwen2.5-7b-instruct") sets the model name used for inference. This string must match the model name that llama-server reports in its model list.

The triage system maps queries to ideal model families:

  • code → models with "coder" or "qwen" in the name
  • general → models with "mistral" or "llama" in the name

For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query.

Verifying the Installation

# Check the app is running
curl http://localhost:8080/

# Check health endpoints
curl http://localhost:8080/api/hardware
curl http://localhost:8080/api/models
curl http://localhost:8080/api/cluster

# Get a guest session
curl -X POST http://localhost:8080/api/auth/guest \
  -H "Content-Type: application/json" \
  -d '{}'

Security Checklist

  • Set a strong CAIC_ADMIN_PIN (not 1234, not your birthday)
  • Set CAIC_ALLOW_DEFAULT_PIN=false after first login
  • Generate a strong CAIC_COMPLETIONS_API_KEY
  • Review CAIC_ALLOWED_CIDRS — defaults allow all RFC1918 space
  • Set CAIC_TRUSTED_ORIGINS if accessing from non-LAN origins
  • Put cAIc behind a reverse proxy (Caddy, nginx) for HTTPS if exposed beyond LAN
  • Change RabbitMQ password from default
  • Enable CAIC_TRUST_X_FORWARDED_FOR=true if behind reverse proxy

Troubleshooting

"Origin check failed" (403)

cAIc requires either an Origin or Referer header on all /api/ requests. Browser requests include these automatically. For curl:

curl -H "Origin: http://localhost:8080" ...

"Client IP not allowed" (403)

Your IP is not in the allowed CIDR list. Check CAIC_ALLOWED_CIDRS:

# Temporarily allow all (dev only)
export CAIC_ALLOWED_CIDRS="0.0.0.0/0,::/0"

"Rate limit exceeded" (429)

Wait for the rate window to reset (default 60s) or increase limits in config.py.

"Authentication required" (401)

Get a guest session first:

curl -X POST http://localhost:8080/api/auth/guest \
  -H "Content-Type: application/json" \
  -d '{}' \
  -H "Origin: http://localhost:8080"

"Admin PIN required for this action" (403)

Log in as admin first:

curl -X POST http://localhost:8080/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"pin": "1319"}' \
  -H "x-session-id: YOUR_SESSION_ID"

RabbitMQ connection refused

systemctl status rabbitmq-server
ss -tlnp | grep 5672
rabbitmqctl list_users

Worker not showing in cluster status

# On the worker
journalctl -u caic-node-agent --no-pager -n 50

# Verify RabbitMQ reachability
nc -zv 192.168.50.108 5672

# On the coordinator
curl http://localhost:8080/api/cluster

Qdrant connection failed

curl http://localhost:6333/healthz

Database errors

ls -la caic.db
# Nuclear option (data loss!)
rm caic.db
# Restart cAIc — init_db() recreates tables

Logs

journalctl -t caic --no-pager -n 100
tail -f /var/log/syslog | grep caic

Clone this wiki locally