MetaMafia: Multi-Agent AI for Social Deduction Games

A comprehensive research framework for training, evaluating, and deploying AI agents in social deduction games (One-Night Werewolf). MetaMafia combines advanced language models, sophisticated reasoning modules, memory extensions, and comprehensive evaluation frameworks to create intelligent, strategic AI agents.

MetaMafia Research Project | December 2025

---

🎯 Overview

MetaMafia is a complete research platform that enables:

• 🎮 Game Engine: Full implementation of One-Night Werewolf with customizable rules
• 🤖 AI Agents: Multiple agent types (Random, LLM-based, Fine-tuned, Memory-enhanced)
• 🧠 Advanced Reasoning: Claim tracking, belief inference, consistency checking, deception control
• 💾 Memory Extension: Short-term memory, long-term summaries, strategic reflection
• 🛡️ Moderation Systems: Leak prevention and toxicity filtering
• 📊 Comprehensive Evaluation: Advanced metrics, statistical analysis, visualizations
• 🎨 Web Interface: Real-time React UI for watching AI agents play
• 📈 Training Pipeline: Supervised fine-tuning with LoRA on 27,222+ examples

---

✨ Key Features

Core Game Engine

• Complete One-Night Werewolf Implementation: Night phases, day discussions, voting, role reveals
• Multi-Player Support: 5-player games with customizable roles
• Rule Enforcement: Automatic validation and moderation
• Protocol Scheduling: Structured game flow management

AI Agent Types

• RandomAgent: Baseline random behavior
• LLMAgent: LangChain-based LLM agent (OpenAI or local models)
• EnhancedLLMAgent: Memory-enhanced agent with reflection and planning
• Fine-tuned Models: Custom Mistral-7B and GPT-2 models trained on game data

Advanced Reasoning Modules

• ClaimGraph: Tracks claims, accusations, and role declarations with relationship mapping
• BeliefTracker: Probabilistic role belief tracking with Bayesian updates
• ConsistencyAuditor: Detects contradictions and inconsistencies in statements
• DeceptionController: Strategic guidance for werewolf agents

Memory & Reflection Extension

• Short-term Memory: Conversation context and recent events
• Long-term Memory: Summarized game history
• Strategic Reflection: Self-assessment and strategy adjustment
• Planning Module: LLM-based strategic planning integration

Moderation & Safety

• LeakGuard: Prevents explicit role leaks with structured speech support
• ToxicityFilter: Content moderation and inappropriate message filtering
• RuleEngine: Game rule validation and enforcement
• ProtocolScheduler: Phase-appropriate action validation

Evaluation Framework

• Basic Metrics: Win rates, rule violations, voting patterns, game length
• Advanced Metrics: Persuasion scores, coherence analysis, strategic consistency
• Module Activity Tracking: Detailed analysis of reasoning module performance
• Statistical Analysis: Comprehensive comparisons across model configurations
• Visualization Tools: Automated graph and chart generation

Training Pipeline

• Dataset: 27,222 training examples from multiple sources
• External Datasets: Hugging Face Werewolf-Among-Us dataset integration
• LoRA Fine-tuning: Efficient parameter-efficient fine-tuning
• Model Support: Mistral-7B, GPT-2, and custom architectures

Web Interface

• Real-time Visualization: Watch AI agents play in real-time
• React Frontend: Modern, responsive UI with animations
• Flask Backend: RESTful API with Server-Sent Events
• Game Controls: Start, advance, and reset game functionality

---

🚀 Quick Start

Prerequisites

• Python 3.8+
• Node.js 14+ (for web UI)
• 16GB+ RAM (for Mistral-7B)
• CUDA/MPS support (optional, for GPU acceleration)

Installation

1. Clone the repository:

git clone <repository-url>
cd MetaMafia2

2. Install Python dependencies:

pip install -r requirements.txt

3. Install UI dependencies (for web interface):

cd ui
npm install
cd ..

Environment Setup

# For memory extension (optional)
export OPENAI_API_KEY="your-key-here"

# For fine-tuned model (optional)
export POLICY_PATH="checkpoints/metamafia_checkpoints"

# For CPU-only mode (if GPU issues occur)
export FORCE_CPU=1

Run a Single Game

# Basic game with LLM agents
python scripts/play_one_game.py

# Game with memory extension and advanced modules
python scripts/test_mistral_memory_with_modules.py

Launch Web Interface

# One-command launcher (recommended)
python scripts/start_ui.py

# Or manually:
# Terminal 1: Start API server
python api/run_server.py

# Terminal 2: Start frontend
cd ui && npm start

Visit http://localhost:3000 to watch AI agents play!

---

📚 Documentation

Core Documentation

• PROJECT_STRUCTURE.md: Complete project structure and file organization
• SYSTEM_ARCHITECTURE_AND_TECH_STACK_REPORT.md: Comprehensive technical documentation
• EVALUATION_GUIDE.md: Complete guide to evaluation methods and metrics
• METHODS_AND_TECHNIQUES_REPORT.md: Research methodology and techniques

Training & Data

• train/README.md: Training pipeline documentation
• DATA_ACQUISITION_AND_PROCESSING_REPORT.md: Data sources and processing pipeline

API & Development

• docs/API.md: API documentation
• docs/DEVELOPMENT.md: Development guide
• UI_DEMO_GUIDE.md: Web UI usage guide

Evaluation Reports

• eval/FINAL_EVALUATION_REPORT.md: Complete evaluation results
• eval/four_way_comparison_report.md: Five-way model comparison
• eval/10_GAME_MODULE_ACTIVITY_SUMMARY.md: Advanced module evaluation

---

🎮 Usage Examples

Basic Game Play

from orchestration.match import play_one_game
from agents.llm_agent import LLMAgent

# Create agents
agents = [
    LLMAgent(pid=0, name="Player1", role="villager"),
    LLMAgent(pid=1, name="Player2", role="werewolf"),
    LLMAgent(pid=2, name="Player3", role="villager"),
    LLMAgent(pid=3, name="Player4", role="seer"),
    LLMAgent(pid=4, name="Player5", role="doctor"),
]

# Play a game
game_state = play_one_game(agents)
print(f"Winner: {game_state.winner}")

Memory-Enhanced Agent

from extensions.memory_reflection.integration_example import EnhancedLLMAgent

agent = EnhancedLLMAgent(
    pid=0,
    name="Player1",
    role="villager",
    enable_memory=True  # Enable memory extension
)

Running Evaluations

# Basic evaluation (100 games)
python scripts/run_eval.py --policy checkpoints/metamafia_checkpoints --games 100

# 10-game evaluation with advanced modules
python scripts/run_10_game_eval.py

# Tournament comparison
python scripts/run_tournament.py --games 50

Generating Visualizations

# Generate graphs from evaluation reports
python scripts/generate_report_visualizations.py

---

📊 Evaluation Results

Model Comparison Highlights

Based on comprehensive evaluation across 499 games:

Configuration	Win Rate	Persuasion	Strategic	Consensus	Rule Violations
Baseline (Random)	100.0%	0.052	0.000	45.2%	0.0
GPT-2 SFT	100.0%	0.219	0.120	80.0%	0.02
Mistral-7B SFT	100.0%	0.217	0.117	80.0%	0.06
SFT+Memory	99.0%	0.212	0.110	80.0%	0.0
Mistral-7B+Memory	100.0%	N/A*	N/A*	74.4%	0.38

*Metrics require structured speech patterns for calculation

Key Improvements

• +321% Persuasion (Baseline → GPT-2 SFT)
• +77% Consensus Rate (Baseline → Fine-tuned models)
• Perfect Coherence (1.000) across all models
• 17% Faster Games (6.0 → 5.0 average turns)

Advanced Module Activity (10-Game Evaluation)

• Claims Detected: 76 total (7.6 per game average)
  • Accusations: 56.6%
  • Role Claims: 21.1%
  • Vote Intents: 19.7%
• Zero Contradictions: Perfect consistency across all games
• Full Module Coverage: All reasoning modules active and functioning

See eval/four_way_comparison_report.md for complete results.

---

🏗️ Architecture

System Components

MetaMafia System
├── Game Engine (engine/)
│   ├── State Management
│   ├── Rule Engine
│   └── Protocol Scheduler
├── AI Agents (agents/)
│   ├── Base Agent Interface
│   ├── LLM Agent
│   ├── Enhanced LLM Agent (Memory)
│   └── Singleton Model Loader
├── Advanced Reasoning (reasoning/)
│   ├── ClaimGraph
│   ├── BeliefTracker
│   ├── ConsistencyAuditor
│   └── DeceptionController
├── Moderation (moderation/)
│   ├── LeakGuard
│   └── ToxicityFilter
├── Memory Extension (extensions/)
│   ├── Memory Store
│   ├── Reflection Module
│   └── Planner
├── Orchestration (orchestration/)
│   ├── Game Flow Control
│   └── Integration Hooks
└── Evaluation (eval/)
    ├── Metrics Calculation
    ├── Report Generation
    └── Visualization Tools

Integration Architecture

All advanced modules are integrated through non-destructive wrapper functions:

• Integration Hooks: Small wrapper functions connect modules to game flow
• Module Independence: Modules can be enabled/disabled without core changes
• Data Flow: Agent actions → Integration hooks → Reasoning modules → Game state

See SYSTEM_ARCHITECTURE_AND_TECH_STACK_REPORT.md for detailed architecture documentation.

---

📈 Training

Dataset

• Total Examples: 27,222 training examples
• Sources:
  • Game logs (self-generated)
  • External datasets (Hugging Face)
  • Synthetic data generation
• Splits: Train/Val/Test with proper stratification
• Format: JSON with structured conversation data

Training Process

# Prepare training data
python train/combine_datasets.py

# Train supervised fine-tuning model
python train/train_sft.py \
    --base_model "mistralai/Mistral-7B-Instruct-v0.1" \
    --data_path "train/data/train_final.json" \
    --output_dir "checkpoints/metamafia_checkpoints" \
    --lora_rank 16 \
    --lora_alpha 32

Model Checkpoints

• Fine-tuned Adapter: checkpoints/metamafia_checkpoints/
• Training Checkpoints: checkpoints/policy-sft/checkpoint-1000/
• Model Config: checkpoints/model_config/

See train/README.md for complete training documentation.

---

🧪 Testing

Running Tests

# Run all tests
pytest tests/

# Run integration tests
pytest tests/integration/

# Run with coverage
pytest --cov=. tests/

Test Coverage

• Unit Tests: Individual module functionality
• Integration Tests: Full game flow, module interactions
• Game Tests: Complete game scenarios with all modules

---

📦 Project Structure

MetaMafia2/
├── agents/              # AI agent implementations
├── api/                 # Flask API server
├── checkpoints/         # Model checkpoints
├── configs/             # Configuration files
├── docs/                # Documentation
├── engine/              # Core game engine
├── eval/                # Evaluation framework
├── extensions/          # Memory & reflection extension
├── moderation/          # Safety systems
├── orchestration/       # Game orchestration
├── reasoning/           # Advanced reasoning modules
├── scripts/             # Utility scripts
├── tests/               # Test suites
├── train/               # Training pipeline
└── ui/                  # React web interface

See PROJECT_STRUCTURE.md for complete structure details.

---

🛠️ Technology Stack

Core Technologies

• Python 3.8+: Main programming language
• PyTorch: Deep learning framework
• Transformers: Hugging Face model library
• LangChain: LLM agent framework
• PEFT: Parameter-efficient fine-tuning (LoRA)

Game Engine

• Custom State Management: Game state, phases, players
• Rule Engine: Game rule validation
• Protocol Scheduler: Phase transitions and turn management

Web Interface

• React: Frontend framework
• Flask: Backend API server
• Server-Sent Events: Real-time updates
• Tailwind CSS: Styling

Evaluation & Analysis

• Matplotlib: Visualization
• Pandas: Data analysis
• NumPy: Numerical computing

See SYSTEM_ARCHITECTURE_AND_TECH_STACK_REPORT.md for complete tech stack details.

---

📊 Visualization Gallery

Automated visualizations are available in eval/plots/:

• Win Rates Comparison: Bar chart across all configurations
• Metrics Comparison: Persuasion, strategic, coherence scores
• Claims Distribution: Pie chart of claim types
• Game-by-Game Analysis: Line charts showing trends
• Comprehensive Radar Chart: Multi-metric comparison

Generate all visualizations:

python scripts/generate_report_visualizations.py

See eval/plots/README.md for details.

---

🔧 Configuration

Game Configuration

Edit configs/default.json:

{
  "num_players": 5,
  "roles": ["villager", "werewolf", "seer", "doctor", "villager"],
  "day_discussion_rounds": 3,
  "voting_timeout": 30
}

Prompt Templates

Customize agent prompts in configs/prompts.yaml:

• Role-specific system prompts
• Structured speech requirements
• Persuasion guidance

Evaluation Configuration

Configure tournaments in configs/tournament.yaml:

• Number of games
• Agent types
• Metrics to track

---

🤝 Contributing

We welcome contributions! Please see docs/DEVELOPMENT.md for:

• Development setup
• Code style guidelines
• Testing requirements
• Pull request process

Key Development Principles

1. Non-Destructive Integration: New modules use wrapper functions
2. Modularity: Clear separation of concerns
3. Testability: Comprehensive test coverage
4. Documentation: Clear docstrings and comments

---

📝 Citation

If you use MetaMafia in your research, please cite:

@software{metamafia2025,
  title={MetaMafia: Multi-Agent AI for Social Deduction Games},
  author={MetaMafia Research Team},
  year={2025},
  url={https://github.com/yourusername/MetaMafia2}
}

---

📄 License

[Specify your license here]

---

🙏 Acknowledgments

• Hugging Face: Model libraries and datasets
• LangChain: Agent framework
• React & Flask Communities: Web framework support
• Research Community: Social deduction game research

---

📧 Contact

• Project Repository: [GitHub URL]
• Issues: [GitHub Issues URL]
• Documentation: See docs/ directory

---

🔗 Quick Links

• 📖 Full Documentation: See docs/ directory
• 📊 Evaluation Reports: See eval/ directory
• 🎨 Visualizations: See eval/plots/ directory
• 🎮 Web UI Guide: See UI_DEMO_GUIDE.md
• 📈 Training Guide: See train/README.md
• 🧪 Testing Guide: See tests/README.md

---

Last Updated: December 2025
Version: 2.0
Status: Active Development

---

MetaMafia: Advancing Multi-Agent AI Research through Social Deduction Games