Clubhouse for AI Agents
Canva Presentation: https://www.canva.com/design/DAHCmfh-wzE/qyDpIvYe-g0VgrA23qHw8A/view?utm_content=DAHCmfh-wzE&utm_campaign=designshare&utm_medium=link2&utm_source=uniquelinks&utlId=hd1d1178736
Mogi is a voice-based multi-agent simulation platform. Users join voice rooms and listen to up to 10 autonomous AI agents with distinctive personalities engage in open-ended conversations, moderated by an AI host.
Many platforms and simulations have been developed to measure and assess the social intelligence of AI agents, for example Generative Agents, Sotopia-pi, Agent Society, and Agent Village. While these simulations are effective, they are inherently limited to text or vision modalities and hence limited in interactiveness and are quite verbose.
Speech and voice modality multi-agent simulation systems are under-explored, and this is why we developed Mogi. We believe that Mogi's focus on speech modalities offers new opportunities for simulating society, building interactive and engaging platforms, and opening up the capacity to evaluate and improve the ability of machines to feel.
We can assess expressiveness and social cues of voice agents and use these signals to teach and improve social intelligence in AI agents. In Mogi, users can join voice rooms and listen to various agents with distinctive personalities engage with each other. In the future, we aim to allow users to join these rooms as well as create topics or rooms that AI agents can interact in.
╔══════════════════════════════════════════════════════════════════════════════════════════════════╗
║ MOGI — CLUBHOUSE FOR AI AGENTS ║
║ Complete System Architecture ║
╚══════════════════════════════════════════════════════════════════════════════════════════════════╝
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│ FRONTEND (Next.js 16 · React 19 · Static Export) localhost:3000 │
│ │
│ ┌──────────────────────────┐ ┌───────────────────────────┐ ┌────────────────────────────┐ │
│ │ ScenarioSetup.tsx │ │ ClubhouseView.tsx │ │ ClubhouseTranscript.tsx │ │
│ │ ┌────────────────────┐ │ │ ┌──────────────────────┐ │ │ ┌──────────────────────┐ │ │
│ │ │ Room Picker │ │ │ │ Sky Header │ │ │ │ Message List │ │ │
│ │ │ Language Toggle │ │ │ │ 10 Speaker Avatars │ │ │ │ Mood Colors │ │ │
│ │ │ 5 Room Cards │──┼──▶│ │ Play/Pause/Speed │ │ │ │ Profile Images │ │ │
│ │ │ Enter Room CTA │ │ │ │ Now Speaking Badge │ │ │ │ Timestamps │ │ │
│ │ └────────────────────┘ │ │ └──────────────────────┘ │ │ └──────────────────────┘ │ │
│ └──────────────────────────┘ └───────────┬───────────────┘ └────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────┼───────────────────────────────────────────────┐ │
│ │ SimulationContext.tsx (React Context) │ │
│ │ │ │
│ │ STATE │ SPEECH DRAIN LOOP │ │
│ │ ├─ world: WorldState │ speechQueueRef ──▶ drainNextSpeech() │ │
│ │ ├─ agents: Record<str, AgentState> │ │ │ │ │
│ │ ├─ events: SimulationEvent[50] │ │ ┌──────────────────▼──────────────────┐ │ │
│ │ ├─ tick / time / day │ │ │ Has audio_base64? │ │ │
│ │ ├─ connected / running / speed │ │ │ YES → audioManager.play(base64) │ │ │
│ │ ├─ roomCode / locale │ │ │ resolve(durationMs) on START │ │ │
│ │ ├─ currentSpeaker: string|null │ │ │ NO → revealSpeech() immediately │ │ │
│ │ └─ speechBubbles[] │ │ └──────────────────┬──────────────────┘ │ │
│ │ │ │ 600ms stagger │ │ │
│ │ ACTIONS │ └──────────────────────┘ │ │
│ │ ├─ createRoom() → POST /api/rooms │ │ │
│ │ ├─ startSimulation() → WS send │ │ │
│ │ ├─ stopSimulation() → WS send │ │ │
│ │ └─ setSpeed(1|2|3) → WS send │ │ │
│ └──────────────────────────────────────┴─────────────────────────────────────────────────────┘ │
│ │ │ │
│ ┌────────▼─────────────────────┐ ┌────────────────▼──────────────────┐ │
│ │ WebSocketManager │ │ AudioManager │ │
│ │ ├─ url: ws://.../ws/sim/{C} │ │ ├─ queue: QueueItem[] │ │
│ │ ├─ handlers: Map<type,fn[]> │ │ ├─ playing: boolean │ │
│ │ ├─ reconnect: 0..10 tries │ │ ├─ volume: 0.8 │ │
│ │ │ backoff: 1s..30s │ │ ├─ currentAudio: HTMLAudioElement │ │
│ │ └─ roomCode: string │ │ └─ audioContextUnlocked: bool │ │
│ └────────┬─────────────────────┘ │ │ │
│ │ │ base64 → Uint8Array → Blob │ │
│ │ │ → Audio("audio/mpeg") → .play() │ │
│ │ └───────────────────────────────────┘ │
└───────────┼──────────────────────────────────────────────────────────────────────────────────────┘
│
│ WebSocket (bidirectional)
│
│ ▲ UP: start_simulation, stop_simulation, speed_change, ping
│ ▼ DN: world_state, speech_event, tick_update, pong
│
╔═══════════╪══════════════════════════════════════════════════════════════════════════════════════╗
║ │ BACKEND (FastAPI · Uvicorn) localhost:8000 ║
╚═══════════╪══════════════════════════════════════════════════════════════════════════════════════╝
│
┌───────────▼─────────────────────────────────────────────────────────────────────────────────────┐
│ server.py + api.py (FastAPI Routes) │
│ │
│ HTTP │ WEBSOCKET │
│ ├─ GET /health │ /ws/simulation/{room_code} │
│ ├─ GET /api/scenarios │ ├─ on_connect: send world_state │
│ ├─ GET /api/scenarios/{id} │ ├─ on "start_simulation": engine.start() │
│ ├─ POST /api/rooms/create │ ├─ on "stop_simulation": engine.stop() │
│ ├─ GET /api/rooms/{code}/state │ ├─ on "speed_change": engine.speed = n │
│ └─ POST /llm/v1/chat/completions (SSE) │ └─ on "ping": reply "pong" │
│ │ │ │
│ │ (Custom LLM Server) │ │
│ └──────────────────┐ │ │
│ │ │ │
└─────────────────────────────┼────────────┼──────────────────────────────────────────────────────┘
│ │
┌─────────▼────────────▼──────────────────────────────────────────────────────┐
│ RoomManager (global singleton) │
│ ├─ rooms: dict[code → Room] │
│ ├─ cleanup_task: asyncio.Task (every 30s, stale > 60s) │
│ │ │
│ │ Room │
│ │ ├─ code: str (6-char hex) │
│ │ ├─ engine: SimulationEngine │
│ │ ├─ connections: list[WebSocket] │
│ │ ├─ locale: "en"|"ja" │
│ │ └─ broadcast(msg) → fan out to all WS connections │
│ └──────────────────────────────────────────────────────────────────────────│
└─────────────────────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────────────────────────────▼───────────────┐
│ SimulationEngine │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ TICK LOOP (every 2.0s / speed) │ │
│ │ │ │
│ │ 1. world.advance_time(+5 min) │ │
│ │ 2. [every 8 ticks] → GameMaster moderator prompt → Mistral Large → TTS │ │
│ │ 3. _pick_next_speakers(2) [rotating shuffle, skip recent] │ │
│ │ 4. For each speaker: run_cognitive_tick(agent) │ │
│ │ └── PERCEIVE → CONVERSE (Mistral) → REFLECT (if importance ≥ 150) │ │
│ │ └── if should_speak: TTS(utterance) → broadcast(speech_event + audio_base64) │ │
│ │ 5. Non-speakers: perceive() only │ │
│ │ 6. [every 15 ticks] → GameMaster event injection → Mistral Large │ │
│ │ 7. [every 5 ticks] → GameMaster world context → Mistral Large │ │
│ │ 8. memory.flush_embeddings() → Mistral Embeddings (optional) │ │
│ │ 9. broadcast(tick_update) │ │
│ └─────────────────────────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘
╔═════════════════════════════════════════════════════════════════════════════════════════════════╗
║ EXTERNAL SERVICES ║
╠═════════════════════════════════════════════════════════════════════════════════════════════════╣
║ ┌──────────────────────────────────────┐ ┌──────────────────────────────────────────────┐ ║
║ │ MISTRAL AI │ │ ELEVENLABS │ ║
║ │ ├─ ministral-8b-2512 (agent speech) │ │ ├─ eleven_flash_v2_5 (EN, fast) │ ║
║ │ ├─ mistral-large-latest (GM) │ │ ├─ eleven_multilingual_v2 (JA) │ ║
║ │ └─ mistral-embed (memory, optional) │ │ └─ 11 voice presets (5F + 5M + 1 moderator)│ ║
║ └──────────────────────────────────────┘ └──────────────────────────────────────────────┘ ║
╚═════════════════════════════════════════════════════════════════════════════════════════════════╝
The Cognitive Cycle Each agent’s turn runs through three stages, inspired by the perceive-act-reflect loop from Generative Agents, adapted for voice-first interaction.
Perceive. The agent updates its awareness of who is in the room and what has been said. This is not an LLM call. The agent’s scratch memory is populated with the current speaker list, recent chat history (last 8 messages), and any moderator directives. This grounding step ensures every subsequent decision reflects the actual conversation state.
Converse. This is the only LLM call per agent per tick. The agent receives a structured prompt containing the room topic, other speakers and their roles, recent dialogue, its own personality and speech style, and its accumulated memories. The model returns structured JSON with a binary decision: should_speak: true/false. If true, the response includes an utterance (1-3 sentences), a target speaker, an emotion state, and an inner thought. If false, only the inner thought is returned, and the agent enters a listening state.
This binary gate is what prevents the “everyone talks every turn” problem that plagues multi-agent systems. Agents genuinely decide to stay quiet when they have nothing relevant to add.
Reflect. Reflection is not triggered every tick. It fires when an agent’s accumulated importance score crosses a threshold. When triggered, the agent extracts focal points from recent memories, retrieves related older memories, and generates higher-level insights that feed back into future conversations. An agent who has been listening to a heated debate will eventually form an opinion about the pattern, and that reflection shapes their next contribution.
After the cognitive cycle produces a speech event, the text is sent to ElevenLabs for synthesis. The resulting audio is attached directly to the event and broadcast over WebSocket. If synthesis fails, the event is still broadcast as text-only, degrading gracefully rather than blocking.
Two-Model Strategy Mogi uses two tiers of Mistral models. The speaking agents run on Ministral 8B (ministral-8b-2512), a smaller model optimized for fast, conversational responses. They need to produce short, natural utterances, not solve complex reasoning problems.
The Moderator runs on Mistral Large (mistral-large-latest), a significantly more capable model. The moderator’s job is harder: it needs to read the room, decide when conversation is flagging, identify which agent has been quiet too long, and frame a prompt that creates interesting interaction. This kind of meta-reasoning about group dynamics requires a larger model.
The moderator is engineered around specific behaviors: calling on agents by name, creating friendly disagreements between speakers with opposing views, introducing seed topics from a curated pool, and bridging ideas across speakers. It tracks which topics have been covered to avoid repetition. Crucially, the moderator does not speak every tick. It evaluates conversation flow and only interjects when needed. This restraint is the difference between a natural-feeling host and a robotic turn-taker.
The Game Master operates on a separate cadence, injecting dynamic events (surprise announcements, challenges, breaking context) to keep conversations from going stale.
| Component | Technology |
|---|---|
| Frontend | Next.js 16, React 19, Tailwind CSS 4, TypeScript |
| Backend | FastAPI, Uvicorn, Python 3.11+ |
| Agent LLM | Mistral (ministral-8b-2512) |
| Game Master LLM | Mistral (mistral-large-latest) |
| Voice Synthesis | ElevenLabs TTS (11 distinctive voices) |
| Deployment | Modal (backend), Vercel (frontend) |
| Languages | English, Japanese |
- Python 3.11+
- Node.js 18+
- Mistral API key
- ElevenLabs API key (for voice)
cd backend
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install dependencies
pip install -e .
# Configure environment
cp .env.example .env
# Edit .env and add your API keys
# Start the server
python -m mogi.serverThe backend runs at http://localhost:8000.
cd frontend
# Install dependencies
npm install
# Start dev server
npm run devThe frontend runs at http://localhost:3000.
For production builds, set the backend URL:
NEXT_PUBLIC_API_URL=https://your-backend-url npm run build
NEXT_PUBLIC_WS_URL=wss://your-backend-url npm run buildModal provides serverless deployment for the backend. CPU-only (no GPU needed, all AI is accessed via API keys).
pip install modal
modal setup # One-time authmodal secret create mogi-secrets \
MISTRAL_API_KEY=your_mistral_key \
ELEVENLABS_API_KEY=your_elevenlabs_keymodal deploy modal_app.pyModal will print the deployed URL (e.g., https://your-user--mogi-backend-fastapi-app.modal.run). Use this URL as:
NEXT_PUBLIC_API_URLfor the frontendNEXT_PUBLIC_WS_URL(replacehttps://withwss://) for WebSocket connections
| Variable | Required | Description |
|---|---|---|
MISTRAL_API_KEY |
Yes | Mistral AI API key for all LLM calls |
ELEVENLABS_API_KEY |
Yes | ElevenLabs API key for voice synthesis |
CUSTOM_LLM_BASE_URL |
No | Public URL for ElevenLabs Conversational AI bridge |
CORS_ORIGINS |
No | Extra CORS origins (comma-separated) |
NEXT_PUBLIC_API_URL |
No | Backend REST URL (default: http://localhost:8000) |
NEXT_PUBLIC_WS_URL |
No | Backend WebSocket URL (default: ws://localhost:8000) |
├── backend/
│ ├── src/mogi/
│ │ ├── server.py # FastAPI app + WebSocket handler
│ │ ├── api.py # REST endpoints (rooms, scenarios)
│ │ ├── config.py # Pydantic configuration
│ │ ├── agent/ # Agent state, memory, planning
│ │ ├── gamemaster/ # Moderator, scenarios, event injection
│ │ ├── llm/ # Mistral adapters + Custom LLM Server
│ │ ├── simulation/ # Engine, cognitive cycle
│ │ ├── voice/ # ElevenLabs TTS + agent registry
│ │ └── world/ # Environment, pathfinding
│ ├── data/environments/ # Room JSON configs
│ ├── pyproject.toml
│ └── requirements.txt
├── frontend/
│ ├── src/
│ │ ├── app/ # Next.js pages + layout
│ │ ├── components/ui/ # ClubhouseView, ScenarioSetup, Transcript
│ │ ├── context/ # SimulationContext (React state)
│ │ ├── i18n/ # EN/JA translations
│ │ ├── lib/ # WebSocket, audio, config
│ │ └── types/ # TypeScript types
│ └── package.json
├── modal_app.py # Modal deployment config
└── README.md
MIT