Zero-config UX: rapid-mlx as agent runtime, not just inference server

[raullenchai]

Vision

Today: Rapid-MLX is an inference server. Users must select a model, configure parsers, set flags, then separately configure their agent (Hermes, Cursor, Claude Code, etc.) to point at the server.

Goal: Rapid-MLX is an agent runtime that just works. One command, zero configuration.

Current vs Desired UX

Current (developer-oriented)

```bash

1. Install

pip install rapid-mlx

2. Pick a model (user has to know what to choose)

3. Run with flags

rapid-mlx serve mlx-community/Qwen3.5-35B-A3B-4bit \
--continuous-batching \
--tool-call-parser hermes \
--reasoning-parser qwen3

4. Edit ~/.hermes/config.yaml manually:

model:

provider: custom

base_url: http://localhost:8000/v1

default: mlx-community/Qwen3.5-35B-A3B-4bit

context_length: 131072

5. Now run the agent

hermes chat
```

Desired (user-oriented)

```bash
brew install rapid-mlx
rapid-mlx
```

```
🔍 Detected: M3 Ultra, 192GB RAM, 47GB free disk
🔍 Found agents: Hermes (/.hermes/), Claude Code (/.claude/)
📦 Downloading Qwen3.5-35B-A3B (19GB, ~3 min)...
[████████████████████] 100%
✅ Hermes configured → http://localhost:8000
✅ Claude Code configured → http://localhost:8000
✅ Smart routing: simple → local, complex → Claude Opus (ANTHROPIC_API_KEY detected)

Ready. Open a new terminal and run: hermes chat
```

Design Principles

1. Zero config by default — Every flag we expose today must have a sensible auto-detected default.
2. Detect intent — Either auto-detect installed agents, or ask once on first run (`Which agent are you using? [Hermes/Cursor/Claude Code/None]`).
3. Hardware-aware model selection — Pick the largest model that fits in available RAM with headroom. Don't make users learn quantization.
4. Guided downloads — Show progress, ETA, disk usage warnings.
5. Auto-configure agents — Write to `/.hermes/config.yaml`, `/.cursor/config.json`, etc. (with user confirmation on first write).
6. Smart routing built-in — If `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` is set, automatically enable cloud fallback for complex queries.

Existing Building Blocks

We already have most of the pieces — they just need to be orchestrated:

• Agent Profile system (`vllm_mlx/agents/profiles/*.yaml`): Already has `config_path`, `config_template`, `binary_path`, `recommended_models` fields per agent.
• `rapid-mlx agents --setup`: Already exists for manual agent configuration.
• Model registry / aliases (`vllm_mlx/aliases.json`): 36 model aliases ready.
• Auto parser detection (`vllm_mlx/model_auto_config.py`): Maps model family → parser config.
• CloudRouter (`vllm_mlx/cloud_router.py`): Token-threshold routing already implemented.

Proposed Implementation (high-level phases)

Phase 1: Detect & Recommend (no breaking changes)

Add `rapid-mlx setup` command that:

1. Probes hardware (RAM via `mx.device_info()`, disk via `shutil.disk_usage`)
2. Scans for installed agents (filesystem + binary detection)
3. Recommends a model based on RAM
4. Asks user to confirm, downloads + configures everything

Phase 2: Make `rapid-mlx` (no args) trigger setup

If no model specified and no config exists, run setup wizard. Otherwise serve as today.

Phase 3: Smart routing built-in

If cloud API keys detected (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, etc.), enable smart routing by default. Use a small classifier (rule-based + 1-token local model verdict for ambiguous cases).

Phase 4: Continuous discovery

Auto-detect newly installed agents. Notify user: "Goose detected — want to configure?"

Open Questions

• Should the setup wizard be interactive (TUI) or pure CLI prompts?
• How to handle model upgrades (e.g., Qwen3.5 → Qwen3.6 next year)?
• Should we ship a default "router" classifier, or always ask the local model?
• For agents we don't have profiles for — is "generic OpenAI endpoint" enough?

Success Metric

User who has never heard of MLX or model quantization can:

1. Install Rapid-MLX
2. Run one command
3. Have a working AI assistant in <5 minutes

If they can't, we've failed.

Related

• Issue Test Rapid-MLX with more AI clients and report results #47 (Test Rapid-MLX with more AI clients)
• Issue Add request logging middleware #51 (Request logging middleware)
• Existing agent profile system (`docs/agentic-runtime-design.md`)

[mostalive]

That would make it easier.
I will test qwen3.6 later with rapid-mlx, I started with llamacpp as that is my baseline. It needs Reddit to work, there is a --chat-template-kwargs '{"preserve_thinking": true}' \ to preserve thinking tokens, apparently helps cache and prevent loops. It does seem to make a difference. But I don't know where to find that in rapid-mlx yet.
Asking the user is a good idea. recommend one and give two smaller options. I found the qwen 3.5 35B 8bit quant a tight fit on my 64G macbook pro M3 max. I had to limit the context to make it fit in memory limits. (tried MVP4 on llamacpp, does seem to work with 256K context. not that I use all of it. This is probably waiting a few weeks to all the benchmarks come out). To me some of the smaller recommendations feel better for my machine.
When downloading a model, it is nice to print where the model gets stored. I now seem to have two places where mlx models are stored (iniitally downloaded models with simon willisons 'llm' tool with the mlx plugin).
I am curious about smart routing, I think that is a separate project. I read a thread on reddit today discussing how openai and anthropic are not always doing that well.
for UI I would go with CLI with y/n questions. TUI frameworks seem to not be great for accessibility.

[raullenchai]

Superseded by #150 (zero-config rapid-mlx run <agent>). Same vision, consolidated into one issue.