HAI-3.0

MiLo quantized MoS coding agent — a swarm of tiny, hyper-specialized models wired together by a smart orchestration layer. Each model does one job. Together they target 70B+ coding performance at ~1/20th the cost and ~5× the speed.

This is not a coding assistant wrapper. HAI-3.0 is the model system itself.

Why HAI-3.0 Is Different

HAI-3.0 combines two things nobody has combined before:

1. HAI-2.0's pattern composition system — verified pattern library, spec synthesizer, MCTS execution-guided search. Intelligence in the system, not the weights.
2. LocalCodeAI's compression stack — MiLo 3-bit quantization with low-rank compensators, SparseGPT pruning, HNSW on-device retrieval, MoEKD multi-teacher distillation.

The result: a ~800M active parameter model with a 10,000+ pattern library and HNSW retrieval that fits in 8GB unified memory.

Benchmark Results

Benchmark	HAI-3.0
SWE-bench Lite	40.0%

(Note: The current score is 40.0% based on a 10-task sample run of SWE-Bench Lite. The full 300-task benchmark takes ~4 hours to complete on Apple Silicon.)

Installation

The easiest way to install HAI globally is using uv tool install:

# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install HAI globally
uv tool install .

# Now you can use the `hai` command from anywhere!
hai --open "/path/to/your/project"

Quick Start (macOS — Apple Silicon)

HAI runs fully on Mac with real AI inference — no Linux or NVIDIA GPU needed.

Mode	Backend	Model	Speed	Coding quality
mps (default)	PyTorch MPS	Qwen2.5-Coder-1.5B + LoRA	~6s/gen	Best
mlx	Apple MLX	Phi-3.5-mini + LoRA	~2s/gen	Good
stub	Passthrough	None	instant	Tests only

cd "/Users/Hillel/Coding/AI Stuff/HAI"

# One-command: tests → data → train → live demo
./scripts/run_all_mac.sh

# Set up your shell (add to ~/.zshrc)
export HAI_INFERENCE=mps
export HAI_FIXER_CHECKPOINT="$(pwd)/checkpoints/hai-fixer-sft"
export HAI_ROUTER_CHECKPOINT="$(pwd)/checkpoints/hai-router"

# Fix a real bug (AI generates fix, runs tests, retries if needed)
hai fix main.py "returns wrong tax rate" --repo .

# Classify, build, review
hai route "add rate limiting to the API"
hai build utils.py "add a clamp function" --repo .
hai serve --port 8000

Mac training (already run if you used run_all_mac.sh)

uv sync --extra mac --extra dev
uv run python scripts/bootstrap_training_data.py   # 150+ mutation examples
uv run python train/sft_mac.py --max-steps 150     # Qwen2.5-Coder-1.5B LoRA (~6 min)
uv run python train/mlx_lora.py --iters 150        # optional fast-path MLX adapter

Quick Start (Linux + NVIDIA GPU)

# Install dependencies (core — works on macOS/Linux)
uv sync

# GPU training deps (Linux + NVIDIA GPU only)
uv sync --extra gpu

# Copy environment template
cp .env.example .env

# Classify a request (no GPU needed)
hai route "fix the null pointer in parse_config"

# Fix a bug end-to-end (stub mode — no vLLM required)
hai fix main.py "the function crashes on empty input" --repo .

# Start API server
hai serve --port 8000

# Run the orchestrator demo directly
uv run python engine/orchestrator.py

Architecture

User request
     │
     ▼
┌─────────────┐
│ HAI-Router  │  135M classifier, <30ms
│ (classify)  │  fix | build | refactor | review | docwrite | explain
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  RepoGraph  │  tree-sitter AST index
│ (context)   │  surgical extract: function + callees + callers + tests
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Specialist  │  HAI-Fixer / Builder / Refactor / Reviewer / DocWriter
│  (generate) │  1B–3B MoE, Qwen2.5-Coder base → fine-tuned checkpoints
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Verify Loop │  apply patch → run tests → retry up to 3×
│  (validate) │  pytest | jest | cargo test | go test
└──────┬──────┘
       │
       ▼
  Verified code ✓  (or flagged for human review)

Specialist Pool

Model	Params	Job
HAI-Router	135M	Classify task type, route to specialist
HAI-Fixer	3B MoE (500M active)	Bug diagnosis + patch generation
HAI-Builder	3B MoE (500M active)	Net-new function/module generation
HAI-Refactor	1B dense	Code smell detection + restructuring
HAI-Reviewer	1B dense	Security, style, correctness critique
HAI-DocWriter	500M dense	Docstrings, comments, README generation

Training Pipeline

Phase 0 — Synthetic Data ($0)

# Generate 100k bug-fix training instances
uv run python data/generate.py --workers 4

# Generate 60k routing examples
uv run python data/generate_routing.py --no-api  # fallback mode
# Or with Anthropic API:
ANTHROPIC_API_KEY=... uv run python data/generate_routing.py

Algorithmic bug injection via 6 AST mutation operators:

• Off-by-one errors
• Wrong operator substitution
• Missing null checks
• Wrong variable names
• Missing return statements
• Incorrect exception types

Phase 1 — SFT Warm-up (~6 hours, 1 GPU)

uv run python train/sft.py
# → checkpoints/hai-fixer-sft/

QLoRA on Qwen2.5-Coder-3B (4-bit, r=64, 2 epochs).

Phase 2 — RLVR / GRPO (500 steps)

uv run python train/rlvr.py
# → checkpoints/hai-fixer-rlvr/

Reinforcement Learning with Verifiable Rewards — the test runner is the reward function. Binary: pass=1, fail=0. No human annotators.

Phase 3 — Speculative Decoding Heads

uv run python train/eagle_heads.py
# → checkpoints/eagle_heads/

EAGLE-3 style draft heads for 3–4× inference speedup.

Router Training

uv run python train/router_train.py
# → checkpoints/hai-router/

DistilBERT fine-tuned on routing dataset (~20 min).

API

OpenAI-compatible endpoint:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hai-1.0",
    "repo_path": "/path/to/repo",
    "messages": [{"role": "user", "content": "fix the bug in calculate_tax()"}]
  }'

HAI-native endpoints: POST /fix, POST /review, POST /complete

Evaluation

uv run python eval/humaneval.py --limit 10
uv run python eval/latency.py
uv run python eval/cost.py
uv run python eval/swebench.py --limit 10

Benchmark Results

Benchmark	HAI-1.0	Baseline (70B)
SWE-bench Verified	0.0%	TBD
HumanEval	TBD	TBD
Latency p50 (fix task)	TBD	TBD
Cost per task (cloud)	TBD	TBD
Cost per task (self-hosted)	~$0	TBD

Performance Targets

Component	Target
Router	<30ms (CPU)
AST extraction	<100ms (100k LOC repo)
Specialist generation	<2s p50 (with EAGLE)
Verify loop	<35s total
End-to-end fix task	<40s

Project Structure

hai-1/
├── data/           # Bug injection + routing data generation
├── engine/         # Router, AST context, verify loop, orchestrator
├── train/          # SFT, RLVR, EAGLE heads, router training
├── serve/          # FastAPI + CLI
├── eval/           # HumanEval, SWE-bench, latency, cost
└── tests/          # pytest suite

Train Your Own Specialist

1. Generate domain-specific data with data/generate.py (add repos to data/repos.txt)
2. Update train/configs/sft_config.yaml — set specialist and output.checkpoint_dir
3. Run SFT → RLVR pipeline
4. Point HAI_*_CHECKPOINT env vars to your checkpoint
5. Restart the API server

License

MIT