OpenPilot — AI Personal Computer Assistant & Agent Runtime Platform OpenPilot is a single-process, all-in-one AI agent runtime platform supporting multi-model, multi-channel, and multi-agent collaboration. It provides a complete HTTP/WebSocket API gateway and a Web control panel. Designed as a personal computer assistant, it covers a wide range of daily life and work tasks.
🎯 What It Can Do: Life & Work Task Checklist 📁 File & Document Processing 💻 Shell Commands & System Operations 🌐 Network & Information Retrieval 🧠 Memory & Knowledge Management 💬 Multi-Channel Messaging 🤖 Multi-Agent Collaboration 🔮 PolyOracle — AI Prediction Market Analysis 📊 Quant Copilot — AI-Powered Stock Analysis 📈 Auto Quant Trading — Automated Quantitative Trading Pipeline
Signal Evaluation — Confidence-based filtering (high→0.9, medium→0.6, low→0.3) with configurable thresholdAuto Order Execution — Signals automatically converted to orders via Longport broker APIQuantity Calculation — Supports fixed quantity, fixed amount, and Kelly formula sizing modesStop-Loss / Take-Profit — Automatic monitoring with configurable check intervals, triggers market sell ordersTrade Notifications — Order creation, fill, failure, stop-loss trigger, and risk alerts pushed to Telegram/DiscordPaper / Live Mode — Paper mode routes through Longport's simulated API; live mode uses real account (same codebase, different Access Token)Signal Deduplication — Configurable dedup window prevents duplicate orders for the same signalStrategy Engine — Technical indicator strategy scanning and backtestingPortfolio Management — Position tracking and P&L analysisRisk Control — Pre-order risk checks (position limits, daily loss limits, order size limits)Trading Dashboard — Real-time order list, pipeline status, stop-loss monitors, auto-trading config panelAudit Trail — All trading operations logged with full request/response details
⏰ Scheduled Task Scheduling 🔧 System Configuration & Management
Web Control Panel — Full management interface covering chat, sessions, models, channels, agents, skills, config, cron, usage, audit, etc.Dynamic Configuration — Modify system config in real-time via UI or API, no manual file editing neededField-Level Descriptions — Config UI fetches schema metadata to display labels and help text for each fieldMulti-Model Switching — Supports 11+ AI providers, 30+ models, freely switchable at runtimeAPI Key Rotation — Multiple keys per provider with automatic switching + cooldown mechanismFailover — Automatic fallback to alternative models when the primary model failsAudit Logging — All tool invocations are traceable
🎤 Voice Interaction (STT/TTS) 🎬 Video Editing (Phase 1 MVP)
Scenario
How It Works
Get prediction market opportunities every morning
Cron scheduled scan → AI analysis → Telegram push
Analyze stock technical indicators
Quant Copilot → Python technical analysis → AI signal generation
Remotely operate your computer from phone
Send message via Telegram/Discord → AI executes Shell commands
Organize project files
Describe needs in chat → AI reads/writes files + executes commands
Monitor website changes
Scheduled task + browser screenshot + push notification
Code review & modification
Discuss in chat → AI reads code → generates patches
Query real-time information
AI calls HTTP tools to fetch API data
Manage multiple AI roles
Create different agents, bind to different channels/groups
Remember personal preferences
AI auto-writes to USER.md, auto-loads in next conversation
Generate promotional images
Describe the scene → AI calls image generation → auto-sends to chat
Generate PDF reports
Provide content → AI generates Markdown → auto-converts to PDF
Create PPT presentations
Describe outline → AI generates slide JSON → auto-generates .pptx
Send voice messages to AI
Send voice on Telegram → STT transcription → AI processes → voice reply
Edit video clips
Send video + "trim 1:20-1:40" → AI generates Timeline → FFmpeg renders
Auto quant trading
Stock scan → AI signal → confidence filter → auto order → stop-loss monitor → notification
Paper trading test
Configure paper mode → orders route to Longport simulated API → verify strategy without real money
┌─────────────────────────────────────────────────────────┐
│ Control UI (React) │
│ Vite + TailwindCSS + Zustand │
└──────────────────────┬──────────────────────────────────┘
│ HTTP / WebSocket
┌──────────────────────▼──────────────────────────────────┐
│ Gateway (Express) │
│ REST API · WebSocket Stream · Static Serve │
├──────────────────────────────────────────────────────────┤
│ Pi Agent Runtime │
│ ReAct Loop · Tool Pipeline · Context Guard │
├──────────┬───────────┬───────────┬───────────────────────┤
│ Model │ Session │ Tool │ Channel │
│ Manager │ Manager │ Executor │ Manager │
│ │ │ │ │
│ OpenAI │ SQLite │ File │ Telegram (grammy) │
│ Anthropic│ JSONL │ Network │ Discord (discord.js) │
│ Google │ LRU Cache │ Shell │ Slack │
│ DeepSeek │ │ Browser │ │
│ Ollama │ │ Patch │ CommandLane │
│ Qwen │ │ Memory │ InboundDebouncer │
│ 11+ prov │ │ Screen │ PairingStore │
│ │ │ SubAgent │ │
│ │ │ Polymarket│ │
│ │ │ Stock │ │
│ │ │ Image │ │
│ │ │ Document │ │
│ │ │ Voice │ │
│ │ │ Video │ │
│ │ │ Trading │ │
├──────────┴───────────┴───────────┴───────────────────────┤
│ AgentManager · SubagentRegistry · PolicyEngine · Audit │
├──────────────────────────────────────────────────────────┤
│ CronScheduler · PolymarketScanner · StockScanner │
│ NotificationService · VoiceService · ImageRouter │
│ AutoTradingPipeline · TradingGateway · StopLossManager │
│ SignalEvaluator · StrategyEngine · PortfolioManager │
├──────────────────────────────────────────────────────────┤
│ Sandbox · PluginManager · Skills │
└──────────────────────────────────────────────────────────┘
Layer
Technology
Backend Runtime
Node.js + TypeScript
HTTP Framework
Express 5
WebSocket
ws
Database
SQLite (better-sqlite3) + JSONL audit logs
Frontend Framework
React 19 + Vite 7
Styling
TailwindCSS 4
State Management
Zustand 5
Code Editor
Monaco Editor
Charts
Recharts
Telegram
grammy
Discord
discord.js v14
AI SDK
openai · @anthropic-ai/sdk · @google/generative-ai
Scheduled Tasks
node-cron
Voice Synthesis
edge-tts (Python)
Stock Analysis
yfinance + pandas (Python)
Trading
longport SDK (Longport OpenAPI)
Video Processing
FFmpeg / FFprobe
Backend Testing
Jest + ts-jest + fast-check (property-based)
Frontend Testing
Vitest + Testing Library
Build
tsc (backend) + Vite (frontend)
Feature Completion Status
PolyOracle — AI Prediction Market Analysis Quant Copilot — AI Stock Analysis Auto Quant Trading — Automated Trading Pipeline
Voice Interaction (STT/TTS) Video Editing (Phase 1 MVP) Multi-Channel Multi-Agent Collaboration
openpilot/
├── src/ # Backend source code
│ ├── index.ts # Entry point, full bootstrap flow
│ ├── api/ # Express API gateway + WebSocket
│ │ ├── server.ts # 60+ REST endpoints + WS streaming
│ │ ├── tradingRoutes.ts # Trading API endpoints (orders, config, pipeline, stop-loss)
│ │ └── middleware.ts # Rate limiting, input validation, security
│ ├── runtime/ # AI runtime
│ │ ├── AIRuntime.ts # Core execution engine (retry, failover, concurrency)
│ │ └── sandbox.ts # Sandbox isolation
│ ├── pi-agent/ # Pi Agent ReAct loop
│ │ ├── PiAgent.ts # ReAct main loop
│ │ └── PiSession.ts # Session transcript management
│ ├── models/ # Model providers
│ │ ├── ModelManager.ts # Model discovery, config, rotation, failover
│ │ ├── OpenAIProvider.ts # OpenAI / compatible APIs
│ │ ├── AnthropicProvider.ts
│ │ └── GeminiProvider.ts # Google Generative AI
│ ├── session/ # Session persistence
│ │ ├── SessionManager.ts # SQLite + LRU Cache + JSONL
│ │ └── database.ts # Schema initialization
│ ├── channels/ # Multi-channel system
│ │ ├── types.ts # Channel plugin abstraction layer
│ │ ├── ChannelManager.ts # Unified management + routing + health check
│ │ ├── TelegramChannel.ts # Telegram (grammy)
│ │ ├── DiscordChannel.ts # Discord (discord.js v14)
│ │ ├── SlackChannel.ts # Slack
│ │ ├── CommandLane.ts # Concurrency control lane
│ │ ├── InboundDebouncer.ts # Inbound debouncing
│ │ └── PairingStore.ts # Device pairing
│ ├── agents/ # Agent management
│ │ ├── AgentManager.ts # CRUD + files + identity
│ │ ├── SubagentRegistry.ts # Sub-agent lifecycle
│ │ └── types.ts # AgentInfo types
│ ├── tools/ # Tool system
│ │ ├── ToolExecutor.ts # Executor + hook chain
│ │ ├── PolicyEngine.ts # Policy engine
│ │ ├── auditHook.ts # Audit logging
│ │ ├── toolCatalog.ts # Tool catalog (38+ tools, 16 categories)
│ │ ├── fileTools.ts # File operations
│ │ ├── networkTools.ts # HTTP requests
│ │ ├── shellTools.ts # Shell commands
│ │ ├── browserTools.ts # Browser automation
│ │ ├── patchTools.ts # Code patching
│ │ ├── memoryTools.ts # Persistent memory
│ │ ├── screenTools.ts # Screen capture/recording
│ │ ├── polymarketTools.ts # Polymarket market tools
│ │ ├── stockTools.ts # Stock analysis tools
│ │ ├── imageTools.ts # Image generation (multi-engine)
│ │ ├── documentTools.ts # Document generation (PDF/PPT)
│ │ ├── voiceTools.ts # Voice tools (STT/TTS/status)
│ │ ├── videoTools.ts # Video editing (probe/render)
│ │ └── subAgentTools.ts # Sub-agent invocation
│ ├── cron/ # Scheduled tasks
│ │ └── CronScheduler.ts # Cron scheduler (SQLite persistence)
│ ├── services/ # Business services
│ │ ├── PolymarketScanner.ts # Market scanning + AI analysis
│ │ ├── StockScanner.ts # Stock technical analysis + AI signals
│ │ ├── StrategyEngine.ts # Technical indicator strategy engine
│ │ ├── PortfolioManager.ts # Position tracking + P&L analysis
│ │ ├── SignalTracker.ts # Signal lifecycle management
│ │ ├── NotificationService.ts # Push notifications (Telegram/Discord)
│ │ ├── VoiceService.ts # Voice service (STT/TTS multi-engine)
│ │ ├── ImageRouter.ts # Image generation routing (multi-engine)
│ │ └── trading/ # Automated trading system
│ │ ├── types.ts # Trading type definitions
│ │ ├── tradingSchema.ts # DB schema (orders, stop-loss, audit, etc.)
│ │ ├── AutoTradingPipeline.ts # Signal polling + order orchestration
│ │ ├── TradingGateway.ts # Unified entry point (paper/live routing)
│ │ ├── LongportAdapter.ts # Longport broker API adapter
│ │ ├── SignalEvaluator.ts # Confidence filtering + dedup
│ │ ├── QuantityCalculator.ts # Order sizing (fixed/kelly)
│ │ ├── StopLossManager.ts # Stop-loss/take-profit monitoring
│ │ ├── TradeNotifier.ts # Trade notification formatting
│ │ ├── OrderManager.ts # Order CRUD + status transitions
│ │ ├── RiskController.ts # Pre-order risk checks
│ │ ├── PaperTradingEngine.ts # Local paper engine fallback
│ │ └── PositionSyncer.ts # Position sync with broker
│ ├── skills/ # Skill system
│ │ ├── community.ts # Community skills (ClawHub + SkillsMP)
│ │ └── types.ts
│ ├── config/ # Configuration system
│ │ └── index.ts # JSON5 loading + env var overrides + persistence
│ ├── plugins/ # Plugin system
│ │ └── PluginManager.ts
│ ├── types/ # Core type definitions
│ │ └── index.ts
│ └── logger.ts # Structured logging
├── scripts/ # Python scripts
│ ├── stock_analysis.py # Technical indicator calculation (SMA, RSI, MACD, BB)
│ ├── backtest_engine.py # Strategy backtesting engine
│ └── requirements.txt # Python dependencies (yfinance, pandas)
├── frontend/ # Frontend Control UI
│ ├── src/
│ │ ├── App.tsx # Main app + routing
│ │ ├── components/
│ │ │ ├── chat/ # Chat components (message list, input, tool calls)
│ │ │ ├── charts/ # Chart components (K-line chart)
│ │ │ ├── session/ # Session list
│ │ │ ├── model/ # Model selector
│ │ │ ├── common/ # Common components (confirm dialog, error banner, progress bar)
│ │ │ ├── layout/ # Layout (sidebar, topbar)
│ │ │ ├── views/ # Page views (20+: Chat, Sessions, Trading Dashboard, Auto Trading, etc.)
│ │ │ └── tools/ # Audit log component
│ │ ├── stores/ # Zustand state management (uiStore, tradingStore)
│ │ ├── services/ # API client
│ │ ├── hooks/ # Custom hooks
│ │ └── types/ # Frontend types
│ └── index.html
├── data/ # Data directory
│ ├── sessions.db # SQLite database
│ └── sessions-jsonl/ # JSONL audit logs
├── dist/ # Backend compiled output
└── frontend/dist/ # Frontend build output → dist/control-ui/
Node.js >= 20
npm >= 9
Python >= 3.10 (for stock analysis and voice features; auto-configured on startup)
# Clone the projectcd openpilot
# Install backend dependencies# Install frontend dependenciescd frontend && npm install && cd ..# Copy the environment variable template# Edit .env, configure at least one AI provider API Key:# OPENAI_API_KEY=sk-...# ANTHROPIC_API_KEY=sk-ant-...# GOOGLE_AI_API_KEY=AIza...# (Optional) Configure channel Bot tokens:# TELEGRAM_BOT_TOKEN=...# DISCORD_BOT_TOKEN=...# (Optional) Configure Finnhub API Key for stock analysis:# FINNHUB_API_KEY=...# (Can also be set via the Web UI under Quant Copilot config section)# (Optional) Configure Longport for auto quant trading:# Credentials are configured via the Web UI Trading Config panel# (app_key, app_secret, paper_access_token from https://open.longbridge.com)Advanced configuration uses JSON5 format, located at ~/.openpilot/config.json5:
{
// Channel configuration
channels : {
telegram : { enabled : true , token : "..." } ,
discord : { enabled : true , token : "..." } ,
} ,
// Gateway configuration
gateway : {
port : 3000 ,
bind : "loopback" , // loopback | lan | auto | custom
} ,
// Custom model providers
models : {
providers : {
qwen : {
apiKey : "sk-..." ,
baseUrl : "https://dashscope.aliyuncs.com/compatible-mode/v1" ,
models : [ { id : "qwen3.5-flash" , name : "qwen3.5-flash" } ] ,
} ,
} ,
} ,
// PolyOracle notification config
polymarket : {
notify : {
enabled : true ,
telegram : { chatId : "your-chat-id" } ,
minEdge : 0.10 ,
} ,
} ,
// Quant Copilot config
stockAnalysis : {
finnhubApiKey : "your-finnhub-api-key" ,
watchlist : "AAPL,GOOGL,MSFT,TSLA" ,
signalThreshold : 0.6 ,
} ,
// Voice configuration
voice : {
stt : { model : "qwen3-omni-flash/qwen3-omni-flash" , language : "zh" } ,
tts : { auto : "inbound" , model : "edge/default" , voice : "zh-CN-XiaoxiaoNeural" } ,
} ,
// Video editing configuration
video : {
outputDir : "~/.openpilot/generated/video" ,
renderTimeout : 120000 ,
} ,
} # Build backend# Build frontend (outputs to dist/control-ui/)cd frontend && npx vite build && cd ..# Start the service (single process, includes API + WebSocket + Control UI)After startup:
Control UI: http://127.0.0.1:3000 (browser access)
API Gateway: http://127.0.0.1:3000/api/
WebSocket: ws://127.0.0.1:3000/ws
Health Check: http://127.0.0.1:3000/healthz
# Backend tests" database.test" # Frontend testscd frontend && npx vitest --runSingle Process (Recommended) NODE_ENV=production node dist/index.js [Unit]
Description =OpenPilot AI Agent Platform
After =network.target
[Service]
Type =simple
User =openpilot
WorkingDirectory =/opt/openpilot
ExecStart =/usr/bin/node dist/index.js
Restart =on-failure
RestartSec =5
Environment =NODE_ENV =production
[Install]
WantedBy =multi-user.targetFROM node:20-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY dist/ ./dist/
COPY scripts/ ./scripts/
COPY frontend/dist/ ./dist/control-ui/
EXPOSE 3000
HEALTHCHECK CMD curl -f http://localhost:3000/healthz || exit 1
CMD ["node" , "dist/index.js" ]
Mode
Bind Address
Use Case
loopback127.0.0.1
Local development (default)
lan0.0.0.0
LAN access
auto0.0.0.0
Auto-detect
customCustom
Specify gateway.customBindHost
Category
Endpoint
Description
Health
GET /healthzContainer liveness probe
Health
GET /readyzReadiness probe (includes DB check)
Chat
POST /api/chatSynchronous chat
Chat
WS /wsStreaming chat (WebSocket)
Sessions
GET/POST/DELETE /api/sessionsSession CRUD
Models
GET /api/modelsModel catalog
Agents
GET/POST/PUT/DELETE /api/agentsAgent CRUD
Channels
GET /api/channelsChannel status
Skills
GET /api/skills/community/searchCommunity search
Config
GET/PUT /api/configSystem config read/write
Config
GET /api/config/schemaConfig field metadata (labels, descriptions)
Cron
GET/POST/PUT/DELETE /api/cron/jobsScheduled tasks
Polymarket
GET /api/polymarket/marketsMarket data
Polymarket
GET /api/polymarket/signalsAI signals
Polymarket
POST /api/polymarket/scanManual scan trigger
Stocks
POST /api/stocks/analyzeStock technical analysis + AI signal
Trading
GET /api/trading/ordersOrder list (with filter)
Trading
POST /api/trading/ordersPlace order
Trading
GET/PUT /api/trading/configTrading config (mode, auto-trade, quantity, etc.)
Trading
GET /api/trading/accountAccount info (paper or live)
Trading
GET /api/trading/positionsCurrent positions
Trading
GET /api/trading/pipeline/statusAuto-trading pipeline status
Trading
GET /api/trading/pipeline/signalsRecent processed signals
Trading
GET /api/trading/stop-lossActive stop-loss monitors
Usage
GET /api/usageToken usage statistics
Audit
GET /api/audit-logsTool audit logs
Provider
Models
OpenAI
gpt-5.2, gpt-5.1, gpt-5, gpt-4o, o3, o4-mini, etc.
Anthropic
claude-opus-4, claude-sonnet-4, claude-haiku-4.5, etc.
Google
gemini-3-pro, gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash, etc.
DeepSeek
deepseek-chat, deepseek-reasoner
Qwen
qwen3.5-plus, qwen3.5-flash (via OpenAI-compatible API)
Ollama
Local models (auto-discovered)
OpenRouter / Together / Moonshot / Doubao / MiniMax
Via OpenAI-compatible API
MIT