Voile, the Magic Library

Chat message analysis and knowledge archiving system

Named after the Voile, the Magic Library in Touhou Project

---

Voile ingests QQ and WeChat messages, normalizes them into a unified schema, and compiles them into structured knowledge -- topic maps, persona profiles, decision logs, and link archives -- stored in Obsidian and queryable via gRPC.

Architecture

Real-time Collection:
[NapCatQQ]  [WeFlow]
     |           |
  [Go: QQ gw] [Go: WeChat sync]
          \       /
       [Go: API gateway :8080]
              |
    [Python: AstrBot plugin]  <-- message sink (XAR-16)
              |
              v

Historical Import:
[QQ nt_msg.db] --decrypt--> [scripts/decrypt_qq_db.py]
[WeChat DB]    --export-->  [WeChatMsg]
              |
              v

Unified Pipeline:
    [Python: core.storage]    <-- SQLAlchemy, SQLite/Postgres
              |
     +---------+---------+
     |                   |
[Python agents]     [Rust kernel]
topic / sentiment   TextCleaner gRPC :50051
persona / decision  VectorIndex gRPC :50052
     |
[Obsidian vault]

Project structure

voile/
├── core/          Python: schemas, storage, obsidian writer, analysis agents
├── gateway/       Go: QQ gateway, WeChat sync, unified REST API, link fetcher
├── kernel/        Rust: text cleaner + ANN engine (gRPC)
├── kernel/proto/  voile.proto -- single source of truth for all gRPC interfaces
├── plugins/       AstrBot plugin: astrbot_plugin_voile
├── scripts/       Utility scripts for message import and database operations
│   ├── decrypt_qq_db.py    # QQ database decryption tool
│   ├── decrypt_qq_db.sh    # Bash version
│   └── README.md           # Quick start guide
├── docs/          Architecture and development guides
│   └── QQ_DATABASE_DECRYPT.md  # Complete QQ decryption documentation
└── assets/        SVG logo and static assets

Features & Status

Feature	Description	Status
Real-time Collection	QQ/WeChat message streaming via OneBot/WeFlow	✅ Done
Historical Import	QQ/WeChat local database decryption & import	✅ Done
Storage Foundation	Unified schema, PostgreSQL, SQLAlchemy ORM	✅ Done
Analysis Pipeline	Topic extraction, sentiment, persona tracking	✅ Done
Knowledge Archive	Obsidian vault integration, decision logs	✅ Done
Vector Search	Rust-based ANN engine via gRPC	🚧 In Progress
CI/CD & Monitoring	Automated testing, deployment, observability	🚧 In Progress

Quick start

# 1. Copy env template and set your Obsidian vault path
cp .env.example .env
# edit .env: set OBSIDIAN_HOST_PATH

# 2. Start core stack (Redis, Postgres, Rust gRPC, Go gateway, Python workers)
docker compose up -d

# 3. (Optional) QQ login via Lagrange.OneBot
#    Fill in your QQ number in lagrange/appsettings.json (Uin field)
docker compose --profile qq up -d

# Dev: run tests + linters
pip install -e ".[dev]"
pytest
ruff check core/ plugins/ tests/
mypy core/

Message Sources

Voile supports multiple message ingestion methods:

📱 Real-time Message Collection

QQ (via NapCatQQ)

• OneBot v11 protocol adapter
• Real-time message streaming
• Supports text, images, files, and rich media
• Setup: docker compose --profile qq up -d

WeChat (via WeFlow)

• HTTP API for WeChat messages
• Real-time sync from WeChat desktop client
• Supports individual and group chats
• See WeFlow integration guide

📚 Historical Message Import

QQ Local Database

Import years of QQ chat history from local encrypted database:

# 1. Extract encryption key (Windows only, requires admin)
git clone https://github.com/yllhwa/qq-win-db-key.git
cd qq-win-db-key
.\windows_ntqq_get_key.ps1  # Follow prompts to login QQ

# 2. Decrypt and import to Voile
cd /path/to/voile
python scripts/decrypt_qq_db.py <QQ号> <密钥>

What you get:

• ✅ All private chat messages (C2C)
• ✅ All group chat messages
• ✅ Message metadata (timestamps, senders, types)
• ✅ Ready for Voile analysis pipeline

Technical details: docs/QQ_DATABASE_DECRYPT.md

WeChat History Export

Use WeChatMsg to export WeChat history, then import to Voile.

🔄 Import Pipeline

[Encrypted DB] → [Decrypt Script] → [SQLite] → [Voile Importer] → [PostgreSQL] → [Analysis]

All imported messages are normalized into Voile's unified schema and processed through the same analysis pipeline as real-time messages.

Dependencies

Integrates with existing open-source projects -- no reinventing wheels:

• NapCatQQ -- OneBot v11 QQ adapter
• AstrBot -- plugin host for message events
• WeFlow -- WeChat HTTP API
• WeChatMsg -- WeChat history export

Linear board

XartPro / Voile