Voice-native AI companion that fuses OpenClaw agent runtime with Whissle STT voice metatags to create something no other AI assistant can do: understand not just what you say, but how you feel when you say it.
| Capability | ChatGPT / Alexa / Siri | Meta-CLAW |
|---|---|---|
| Emotion detection | From text (unreliable) | From voice signal (acoustic) |
| "I'm fine" + sad voice | Believes the text | Detects the sadness, responds with empathy |
| Simple commands | Always hits LLM (~2s) | LLM-skip via STT intent (~0ms) |
| Behavioral baseline | None | Tracks your normal, detects anomalies |
| Proactive check-ins | None | "You've sounded stressed for a while" |
| Mood trajectory | None | Improving / stable / declining / volatile |
| Tools | Limited | Full OpenClaw toolkit (web, exec, memory, apps) |
┌─────────────────────────────────────────────────────────┐
│ YOUR VOICE │
└──────────────────────┬──────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ WHISSLE STT (only STT) │
│ Transcript + Emotion + Intent + Age + Gender + Speaker │
│ Speech Rate + Pitch + Pauses + Entities + Diarization │
└──────────────────────┬───────────────────────────────────┘
│ MetaFrame
▼
┌──────────────────────────────────────────────────────────┐
│ META-CLAW (this repo) │
│ │
│ Behavioral Intelligence → baselines, stress, mood │
│ Meta Router → emotion gate, LLM-skip, proactive nudges │
│ Prompt Composer → 10-layer voice-enriched system prompt │
│ Emotional Memory → topic + past emotion recall │
└────────┬─────────────────────────────┬───────────────────┘
│ │
▼ ▼
┌────────────────┐ ┌──────────────────────┐
│ Local Skills │ │ OPENCLAW AGENT │
│ (clock, calc) │ │ (all tools, memory, │
│ 0ms, no LLM │ │ web, exec, apps) │
└────────────────┘ └──────────────────────┘
Whissle = only STT with rich metatags (emotion, intent, age, gender, speaker from audio signal) OpenClaw = all tools, skills, MCP servers, memory, agentic execution Meta-CLAW = the voice-behavioral intelligence middleware that makes OpenClaw emotionally aware
- Python 3.11+
- Node.js 22+ (for OpenClaw)
- Whissle STT API token
- Gemini API key
git clone https://github.com/WhissleAI/meta-claw.git
cd meta-claw
pip install -e ".[dev]"npm install -g openclaw@latest
openclaw setup --non-interactive --accept-risk
openclaw models auth paste-api-key --provider google # paste your Gemini API key
openclaw config set agents.defaults.model "google/gemini-2.5-flash"
openclaw gateway run # start the agent runtimeexport GEMINI_API_KEY="your-key"
export WHISSLE_STT_TOKEN="wh_your-token"
meta-claw # starts on http://localhost:8800Visit http://localhost:8800 — two modes:
- Text Mode: Type messages, pick an emotion to simulate voice metatags
- Ambient Listening: Click the mic, speak naturally. Real-time emotion detection + responses.
cd MetaClawApp/MetaClaw
swift build
open .build/MetaClaw.appEvery voice utterance becomes a MetaFrame with fields from Whissle STT:
MetaFrame(
transcript="I'm feeling stressed about my deadline",
emotion="SAD", # from voice prosody, NOT text
emotion_probs={"SAD": 0.66, "FEAR": 0.15, "NEUTRAL": 0.08},
intent="INFORM", # QUESTION, REQUEST, COMMAND, INFORM, COMPLAIN...
age="18_30",
gender="MALE",
speaker_id=0,
speech_rate=SpeechRate(words_per_minute=192, filler_count=0),
stress_score=1.0, # computed by behavioral intelligence
mood_trajectory="declining",
engagement_level="high",
)Routes each MetaFrame through 4 gates:
- Emotion Gate: ANGRY/SAD/FEAR with >60% confidence → always LLM with empathy-first prompt
- Stress Gate: 3+ sustained high-stress segments → proactive intervention
- Pattern Gate: Clock/calculator queries → instant local response, 0ms, no LLM
- Default: Everything else → OpenClaw agent with full voice-enriched context
- Baseline tracking: EMA-smoothed emotion/speech/pitch baseline per user
- Anomaly detection: Flags deviations >2σ from baseline ("you sound different today")
- Stress score: Composite from negative emotion + pitch variation + speech rate
- Mood trajectory: Sliding window → improving / stable / declining / volatile
Voice-behavioral triggers that fire without being asked:
- Morning greeting: Adapts to your voice energy on first interaction
- Stress check-in: "You've sounded stressed for a while. Want to take a breather?"
- Energy drop: "Your voice energy is lower than usual. Coffee break?"
- Mood recovery: "Hey, you sound better than earlier. Whatever shifted, it's working."
- Speech rate alert: "You're speaking faster than usual. Take a breath."
Memories tagged with voice-behavioral context:
Previous mentions of 'budget':
- Jun 05: angry, stressed (75%)
- Jun 05: happy, calm (20%)
Enables: "Last time we discussed the budget, you sounded stressed."
Every LLM call gets a system prompt composed from:
- Base personality (from
lulu.yml) - Voice adaptation instructions
- Safety boundaries
- Archetype style override (6 types from voice classification)
- User name
- Real-time emotion context from MetaFrame
- Emotional protocol (per-emotion response guidelines)
- Behavioral baseline comparison
- Morning greeting context
- Ambient context (memories, calendar, tasks)
| Endpoint | Method | Description |
|---|---|---|
/ |
GET | Demo UI with text + ambient listening modes |
/chat |
POST | Full pipeline: route → skill or agent → response |
/ws/voice |
WS | Ambient voice: stream PCM → real-time responses |
/route |
POST | Route only (for testing) |
/skill |
POST | Direct skill execution |
/skills |
GET | List registered local skills |
/metrics |
GET | Routing metrics (skip rate, latency, action distribution) |
/behavioral/{user_id} |
GET | Behavioral session summary |
/memory/{user_id}/weekly |
GET | Weekly emotional patterns |
/memory/{user_id}/topic/{topic} |
GET | Emotional context for a topic |
/openclaw/status |
GET | OpenClaw availability + agent info |
/health |
GET | Server health check |
meta_claw/
├── schema.py # MetaFrame dataclass + STT segment parser
├── router.py # 4-gate Meta Router with LLM-skip
├── behavioral.py # Baseline tracking, anomaly detection, stress, mood
├── proactive.py # 7 voice-behavioral trigger types
├── emotional_memory.py # Emotion-tagged memory store + pattern analysis
├── prompt_composer.py # 10-layer dynamic system prompt
├── agent.py # Agent runtime (OpenClaw primary, LLM fallback)
├── openclaw_client.py # OpenClaw CLI integration
├── llm_client.py # Direct Gemini/Claude client (fallback)
├── stt_consumer.py # Whissle STT WebSocket consumer
├── server.py # FastAPI server + demo UI
├── config.py # Configuration
├── skills/ # Local LLM-skip skills (clock, calculator)
└── agents/lulu.yml # Character file with emotional protocols
MetaClawApp/MetaClaw/ # macOS SwiftUI app
├── Sources/
│ ├── MetaClawApp.swift
│ ├── Services/VoiceSession.swift # Mic capture + WebSocket
│ └── Views/ContentView.swift # Chat UI + ambient mode
tests/ # 74 unit tests + E2E tests with real STT
# Unit tests (74 tests, ~0.2s)
pytest tests/
# Live STT probe (sends TTS audio through real Whissle STT)
python tests/live_stt_probe.py "What is the weather in San Francisco?"
# Full E2E (TTS → Whissle STT → MetaFrame → Router → LLM response)
GEMINI_API_KEY=... python tests/e2e_stt_to_metaclaw.py
# Complex scenarios (baseline anomaly, mood arcs, emotional memory, archetypes)
GEMINI_API_KEY=... python tests/e2e_complex.py-
Whissle STT metatags are extracted by a custom Conformer model — emotion/intent/age/gender come from acoustic encoder auxiliary heads, not text analysis. Proprietary and compounds with data.
-
Behavioral baselines require time-series voice data. Even with good emotion detection, you need 30 days of voice history to know what "unusual" means for a specific user.
-
LLM-skip only works with audio-level intent classification. Text-based intent detection needs the LLM. Audio-based intent runs in the STT layer — no LLM needed.
-
OpenClaw provides 27+ real tools (web search, exec, memory, cron, image generation, sessions) that the agent can actually use, not just talk about.
MIT