🦊 VirtualClaw

Your AI. Your Character. Your World.
An open-source AI companion with a virtual avatar that talks, reacts, and gets things done.

VirtualClaw combines a live virtual character with a powerful AI agent backend — bridging the expressiveness of projects like AIRI/moeru-ai with the agent capabilities of OpenClaw, wrapped in a character system inspired by SillyTavern.

---

✨ What It Does

• 🎭 Virtual avatar that reacts with real emotions (happy, thinking, focused, surprised...)
• 💬 Natural conversation via streaming WebSocket connection
• 🔀 Multi-LLM support — Claude, OpenAI GPT, or local Ollama models
• 🛠️ Agent-ready architecture — tools and skills coming in Phase 2
• 🔒 Self-hosted, privacy-first — your data stays on your machine

---

🏗️ Architecture

virtualclaw/
├── apps/
│   └── web/                  # React frontend (avatar + chat UI)
└── packages/
    └── agent-gateway/        # Node.js backend (WebSocket + multi-LLM)

Flow:

User → Chat UI → WebSocket → Agent Gateway → LLM Router → Claude / OpenAI / Ollama
                                    ↓
              Emotion Detector → Avatar animation state

---

🚀 Quick Start

Prerequisites

• Node.js 20+
• At least one LLM API key (or Ollama running locally)

1. Clone & Install

git clone https://github.com/Kiendt91/virtualclaw.git
cd virtualclaw
npm install

2. Configure Environment

cd packages/agent-gateway
cp .env.example .env
# Edit .env and add your API key(s)

LLM_PROVIDER=claude          # claude | openai | ollama
ANTHROPIC_API_KEY=sk-ant-... # for Claude
OPENAI_API_KEY=sk-...        # for OpenAI
OLLAMA_BASE_URL=http://localhost:11434  # for Ollama

3. Run (Development)

# From root — starts both gateway and web app
npm run dev

Or start separately:

npm run dev:gateway   # http://localhost:3001
npm run dev:web       # http://localhost:5173

4. Open

Visit http://localhost:5173 and start talking to your AI companion!

---

🐳 Docker

cp packages/agent-gateway/.env.example .env
# Fill in your API keys in .env
docker compose up

App available at http://localhost

---

🔌 WebSocket API

Connect to ws://localhost:3001/ws

Send a message

{
  "type": "chat",
  "id": "msg_001",
  "content": "Hello!",
  "history": [],
  "options": {
    "provider": "claude"
  }
}

Response events

Event	Description
chat_start	Stream begins, returns provider + model
chat_chunk	Streamed text delta
chat_end	Full response + emotion tag
error	Something went wrong
status	Current server config

Emotion tags

The gateway automatically detects and returns one of: idle · happy · excited · thinking · focused · surprised · proud · sad

---

🗺️ Roadmap

• Phase 1 — MVP ← We are here

  • Agent Gateway with WebSocket streaming
  • Multi-LLM support (Claude, OpenAI, Ollama)
  • Emotion detection system
  • React frontend with animated avatar placeholder
  • Docker Compose setup

• Phase 2 — Character System

  • Live2D / VRM avatar renderer
  • Character card system (SillyTavern-compatible import)
  • Persistent memory (SQLite)
  • Lorebook / world info

• Phase 3 — Agent Power

  • Tool execution (file, shell, browser)
  • "Working" avatar state during task execution
  • Telegram / Discord channel support

• Phase 4 — Desktop & Community

  • Tauri desktop app
  • Community character card repository
  • Plugin SDK

---

🤝 Contributing

All skill levels welcome! See what's needed:

Area	Skills	Good first issue
Avatar Engine	Three.js, Live2D SDK	Replace emoji with VRM model
Backend	Node.js, TypeScript	Add new tool (e.g. weather API)
Frontend	React, CSS	Improve chat UI animations
AI/ML	Python	Better emotion detection model
DevOps	Docker	GitHub Actions CI/CD

1. Fork the repo
2. Create a feature branch: git checkout -b feat/my-feature
3. Commit your changes
4. Open a Pull Request

---

📄 License

MIT — use it, fork it, build on it.

---

Made with ❤️ for the open-source AI community