Skip to content

bejranonda/ExternalBrain

Repository files navigation

Brain Platform

The knowledge you build while coding with any AI tool, made permanent, portable across tools, and compounding over time.

Brain Platform is an MCP server + webapp that ingests vibe-coding sessions from any AI coding tool (Claude Code, Cursor, Windsurf, Autobahn, custom agents), extracts durable knowledge, and serves it back as typed skills, rules, and a conversational Oracle.

  • No vibe-coding inside. We are the substrate, not another AI coding tool.
  • MCP-first. Every AI tool is a potential client.
  • Multi-tenant. Personal brains, team brains, community pool.
  • Self-improving. Autoskill, evolution, and internal wisdom skills make the brain sharper over time.

How it feels to use

Brain is deceptively simple: a clean surface that opens into deep functionality the moment you need it.

  • The dashboard opens with a single-sentence pulse line ("Your brain knows 42 things from 7 sessions across 3 projects") and three big action cards: how many skills you have, the latest thing the Brain learned, and a one-tap prompt to ask the Oracle. Everything else — projects, sessions, system chatter — sits behind a ▾ Show everything fold. When the Brain extracts new knowledge while you work, a quiet "Brain learned" toast surfaces it.
  • The sidebar collapses to icons by default and expands on hover; every nav destination opens with a plain-English subtitle under its heading so first-time visitors aren't left guessing what a screen is for.
  • First-run users land on /welcome: pick your AI tool → copy the install command → wait for your first session. The EmptyBrainCallout on the dashboard links back into it any time.
  • The Oracle looks like a chat. Behind every answer is a typed retrieval over your own knowledge graph.
  • The skills panel looks like a settings page. Behind it is an autoskill loop that proposes, you approve, and the brain compounds.

We call this progressive disclosure — show what you need now, keep the power one click away. The UI stays out of your way until you ask it for more. Full rules for contributors in docs/DESIGN_PRINCIPLES.md.


AI Application

Read this first

Doc Why
docs/END_USER.md If your team has set up a Brain — sign in, wire Claude Code / Cursor, ask the Oracle, teach a rule
docs/HOW_IT_WORKS.md Step-by-step walkthrough with examples — sign-up → token → first session → KEA extraction → KRA retrieval → Oracle answer → effectiveness loop
docs/QUICKSTART.md If you're installing the Brain locally — zero to a running stack in 15 minutes
docs/DEPLOY_CHECKLIST.md If you're deploying to a public-internet VM — step-by-step pilot checklist (DNS, OAuth, vouchers, smoke) — covers both dev Brain (HZ-Agent) and prod Brain (Autobahn Bot)
docs/HZ_AGENT_MIGRATION.md Relocating or rebuilding the dev Brain host — Cloudflare DNS swap, fresh-host provisioning, post-deploy smoke (operator runbook)
docs/CONTRIBUTING.md Contributing code — branch model (develop runs on VPS-Agent dev Brain; main runs on Autobahn Bot prod Brain), three workflows (Feature feature/* → develop, Bug bugfix/* → develop, Hotfix hotfix/* → main + back-merge), PR conventions, env discipline
AGENTS.md (= CLAUDE.md = GEMINI.md) AI assistants working in this repo — workflow selection, hard rules (no direct push to main/develop, no SSH to hosts, no real secrets), tier-1 reading order, repo-layout cheatsheet
docs/BLUEPRINT.md Product thesis, architecture, reference-system adaptations, business model
docs/STATUS.md What's actually working today (backend / frontend / auth / tests)
docs/APPROACH.md Philosophy, method, decision framework
docs/KNOWLEDGE.md Knowledge representation, ontology, lifecycle, invariants (normative)
/docs/concepts/vocabulary (deployed) · docs-content.ts Locked glossary — the five user-facing words (Brain · Skill · Session · Oracle · Proposal) + advanced acronyms (KEA / KRA / MCP / SQS). Read before writing user-visible copy or new docs.
docs/ARCHITECTURE.md Deployment topology (two Brains, two hosts, shared code via develop → main) + 3-layer / 8-subsystem diagram + data flow — canonical source for which host does what
docs/MCP_TOOLS.md The 12 brain_* MCP tools (incl. brain_create_project / brain_list_projects / brain_get_active_project) + 4 resources
docs/ROADMAP-2026-05-25.md Opinionated 5-day priority for what comes next after v0.14.0 — read before picking up the next slice
docs/CLIENTS.md Wiring Claude Code / Cursor / Windsurf to the HTTP transport
docs/USING_BRAIN.md Daily workflow with Claude Code (and similar) — install one-liners, verification commands, trigger-phrase map, real session walkthrough, debug recipes, operator habits that compound
docs/DESIGN_PRINCIPLES.md UI design principles — progressive disclosure + earned surface area + quiet-by-default. Read before adding a panel, tab, or empty-state. Terse version in GUIDELINES.md §10a.
deploy/DEPLOY.md One-script Docker deploy for a test server
deploy/PRODUCTION.md Bare-VM production deploy with Caddy + auto-TLS
docs/RUNBOOK.md Symptom table + recovery playbooks for production
docs/DEV_PLAYBOOK.md Testing, debugging, and feedback recipes for the develop Brain — counterpart of the prod RUNBOOK
docs/REST_API.md REST endpoints
docs/USECASES.md Doctor/secretary analogies + autoskill
docs/ROADMAP.md Phase-by-phase plan with gates
docs/BUSINESS.md Revenue models + wild-card opportunities
docs/SECURITY.md Auth modes, voucher-code registration, admin surface, zero-error release loop
docs/VALIDATION.md Validation methodology placeholder — previous benchmarks retired with the demo seed (2026-05-08)
docs/KNOWN_ISSUES.md Risks, stop-conditions, open questions
docs/GUIDELINES.md Code style, testing, PR process
docs/NAVIGATION.md Webapp nav iteration process (zero-error checklist)
docs/WIRING.md How a GUI surface wires to the backend (4-file contract)
docs/SYNC.md Optional LiveSync bridge for Obsidian
docs/README.md Index + reading orders
research/knowledge/README.md ~7,900 lines of prior analysis (four passes)
research/autoskill/spec.md nicknisi/autoskill captured spec — adopted in packages/core/src/autoskill.ts

Illustrations

Visual diagrams explaining how the platform works are available in docs/assets/illustrations/:

Diagram What it shows
User Flow (source) How AI coding tools connect to the Brain Platform via MCP and how knowledge is extracted into Skills and Rules
Architecture (source) 3-layer system architecture: Clients → Frontend → Core Services → Database
AI Application (source) Where LLMs and vector embeddings are used: KEA extraction, KRA retrieval, Oracle chat, Autoskill
Process Logic (source) End-to-end session lifecycle: retrieve → log → report → extract → evolve
Vibe-Coding Improvement (source) How Brain Platform acts as a bridge to improve vibe-coding quality and performance
Skill Development (source) The pipeline of how the platform extracts and refines skills and knowledge from coding sessions
Knowledge Algorithm (source) Sequential state algorithm for knowledge logging, analysis, consolidation, and evolution

Each diagram has a .md source file (Mermaid) and a rendered .png. To regenerate after edits: npx @mermaid-js/mermaid-cli -i <file>.mmd -o <file>.png.

Monorepo layout

/apps
  /web          Next.js webapp (dashboard, Oracle chat, skills browser, settings)
  /mcp-server   TypeScript MCP server — stdio + HTTP transport
  /worker       Background jobs: KEA extraction, decay, evolution, autoskill
  /sync-bridge  Optional CouchDB-based LiveSync bridge for Obsidian vaults
/packages
  /types        Shared TypeScript types (Knowledge, Session, Skill, GraphEdge)
  /db           Prisma schema + client
  /core         Intelligence Layer (KEA, KRA, Oracle, autoskill, graph, evolution)
  /mcp-tools    Shared MCP tool definitions (used by mcp-server AND webapp API)
  /sdk-js       Customer SDK for integrators

Quickstart

Full walkthrough (15 min, install → wired Claude Code → first Oracle answer): docs/QUICKSTART.md.

Two templates — pick one:

git clone https://github.com/bejranonda/ExternalBrain.git external-brain && cd external-brain

# A) Local dev / solo machine: dev-shim auth, works without a GitHub OAuth app
cp .env.example .env

# B) Pilot / multi-user: OAuth + voucher gate + admin promotion — REQUIRED for any public URL
cp .env.pilot.example .env
# Then fill in AUTH_GITHUB_ID / AUTH_GITHUB_SECRET (create at https://github.com/settings/developers),
# AUTH_SECRET (openssl rand -base64 32), ADMIN_EMAILS (your email), and one LLM key.

./scripts/deploy.sh                          # build, migrate, seed, backfill, up — idempotent
./scripts/verify-lockdown.sh                 # audit runs automatically, re-run any time
# Webapp:  http://localhost:3000  →  /admin/vouchers  →  issue codes to pilot users
# MCP HTTP: http://localhost:3100/mcp

The deploy scripts run verify-lockdown.sh automatically at the end and print PASS/FAIL per auth-mode expectations. deploy-prod.sh refuses the deploy on a lockdown failure. For troubleshooting + tool wiring snippets: docs/QUICKSTART.md §7. For a production bare-VM deploy with TLS: deploy/PRODUCTION.md. For the auth threat model: docs/SECURITY.md.

Which script to run when

After the first deploy, day-two operations (reload one service, re-run a migration, audit auth, back up/restore) each have a smaller command. The decision table lives in docs/CONTRIBUTING.md §"Which script to run when".

Local dev (no Docker):

# Prerequisites: Node 20+, pnpm 9+, Postgres 16 with pgvector
pnpm install
cp .env.example .env                              # fill in DATABASE_URL + one embedding key (GOOGLE_GEMINI_API_KEY recommended)
psql $DATABASE_URL -c "CREATE EXTENSION IF NOT EXISTS vector;"
pnpm --filter @brain/db prisma migrate deploy     # uses the committed initial migration
psql $DATABASE_URL -f packages/db/sql/session-fts-index.sql   # FTS GIN indexes (idempotent)
pnpm --filter @brain/worker backfill:embeddings   # one-shot; worker schedule picks up new rows
pnpm turbo run dev                                # webapp + mcp-server + worker in parallel
# Webapp: http://localhost:3000 — empty until you sign in and start a session.

Wire an AI tool: new users land on /welcome after sign-in — pick your AI tool, copy the install command, and watch the dashboard pulse line update on your first session. From the user menu the same surface is reachable via MCP tokens → Create; the post-mint wizard generates a one-line installer for your OS + client. For Claude Code on POSIX:

curl -fsSL https://<your-host>/api/onboard.sh | bash -s 'bp_…'

The v2 installer (2026-05-11) runs claude mcp add, then smoke-tests the round-trip through your network → Caddy → MCP server (initialize + tools/call brain_get_user_style), then sends an install-ping (brain_start_session → brain_log_event → brain_report_session_outcome) so KEA gets a real Session row from day 0. A failed install fails loud with per-HTTP-code diagnostics. A successful install prints a literal first-touch prompt you can paste into Claude Code to immediately fire brain_teach_knowledge. Full walkthroughs in docs/CLIENTS.md; MCP tool reference + log-op diagnostics in docs/MCP_TOOLS.md.

KEA provider config (2026-05-12). Knowledge extraction needs an LLM API key matching the configured KEA_MODEL:

  • KEA_MODEL=qwen3-coder (default, cheapest ~$0.001/extraction) → set DASHSCOPE_API_KEY
  • KEA_MODEL=claude-haiku-4-5 (~$0.01/extraction, no new key if you already have Anthropic) → uses ANTHROPIC_API_KEY
  • KEA_MODEL=gpt-4o-mini (~$0.003/extraction) → uses OPENAI_API_KEY

If knowledge_by_kea stays at 0 even after closed sessions, see docs/RUNBOOK.md §"Why is knowledge_by_kea stuck at 0?". For the daily cross-session compounding pipeline (2026-05-14), see docs/RUNBOOK.md §"Cross-session KEA didn't run on its schedule" — it runs automatically at 0 6 * * * UTC and skips users with no new sessions.

Core tech choices (defaults — confirm in docs/BLUEPRINT.md §14)

Concern Default
Runtime Node 20 LTS
Webapp Next.js 16, React 19, Tailwind 4
DB Postgres 16 + pgvector
Embeddings Provider-agnostic via EMBEDDING_BASE_URL — any OpenAI-compatible endpoint. Tested: Google gemini-embedding-001 (free tier, 1536 dim) · DashScope Qwen3 text-embedding-v4 (1536 dim) · OpenAI text-embedding-3-small (default). Swap by flipping one env var; .env.example has three ready blocks.
KEA model Qwen3-Coder (DashScope) or Claude Haiku 4.5 or GLM 4.5
Oracle model Claude Sonnet 4.6 or GLM 5.1 via ANTHROPIC_BASE_URL=https://api.z.ai/api/anthropic
Background jobs pg-boss (no Redis required)
Monorepo Turborepo + pnpm workspaces
Auth NextAuth v5
MCP SDK @modelcontextprotocol/sdk

Status

The Brain is pilot-ready and actively used day-to-day. For the curated milestone table and the full release history (v0.1 → v0.15.0), see docs/STATUS.md. Tagged releases (from v1.0) live on the GitHub Releases page.

Surfaces, transports & background jobs

Six surfaces — primary endpoints

Surface Primary endpoint(s)
Dashboard GET /api/dashboard, GET /api/dashboard/live
Oracle POST /api/oracle/stream (SSE, default), POST /api/oracle, POST /api/knowledge/retrieve, POST /api/oracle/feedback
Skills GET /api/knowledge, PATCH /api/knowledge/[id] (metadata only), DELETE /api/knowledge/[id], POST /api/knowledge/[id] (fork), GET /api/export/rules
Graph GET /api/graph, GET /api/knowledge/[id]/related
Autoskill GET /api/autoskill/proposals, POST/PATCH /api/autoskill/proposals/[id]
Sessions GET /api/sessions (cursor-paginated, outcome+client filters); MCP brain_session_search (Postgres FTS)

All /api/* traffic passes through apps/web/proxy.ts, which enforces per-IP sliding windows from the existing RATE_LIMIT_* env vars and emits x-ratelimit-* response headers.

Auth is handled by getCurrentUserId() in apps/web/lib/brain/auth.ts — NextAuth v5 session (Credentials or GitHub OAuth). Four mutually-exclusive modes: CREDENTIALS (default pilot), OAUTH, DEV_SHIM (ALLOW_DEV_AUTH=true), UNCONFIGURED (503). See docs/SECURITY.md for the threat model and docs/WIRING.md for the full wiring matrix.

Build note: apps/web uses webpack (not Turbopack) due to NodeNext .js.ts resolution. The package.json build script already passes --webpack; pnpm build works without flags.

The frontend component tree lives under apps/web/components/brain/ and its design tokens under apps/web/app/globals.css. Default language is English; Thai and German dictionaries ship as scaffolds and activate via the Tweaks panel. Tweaks server-side sync requires a new Prisma model and is not yet shipped (needs DB migration approval).

Transports

  • MCP stdio (default): BRAIN_MCP_TOKEN=… pnpm --filter @brain/mcp-server dev — wire into Claude Code / Cursor / Windsurf.
  • MCP HTTP (Streamable): MCP_TRANSPORT=http MCP_SERVER_HTTP_PORT=3100 pnpm --filter @brain/mcp-server dev — clients pass Authorization: Bearer <token> against POST /mcp. A health probe is at GET /health.
  • Oracle SSE: POST /api/oracle/stream yields event: delta frames as the model generates, followed by a single event: final carrying citations, confidence, related questions, and token usage.

Background jobs (pg-boss)

Job Schedule Handler
evolution.decay daily 02:00 @brain/core/evolution
evolution.consolidate daily 03:00 @brain/core/evolution
evolution.detect-obsolescence weekly Sun 04:00 @brain/core/evolution
evolution.health-snapshot weekly Sun 05:00 @brain/core/evolution
embeddings.backfill every 10 min apps/worker/src/backfill-embeddings.ts
kea.extract on demand @brain/core/kea
autoskill.run on demand @brain/core/autoskill

Before writing production code, confirm the 7 open decisions in docs/BLUEPRINT.md §14 (build path, LLM providers, embedding model, pricing, data residency, OSS scope, community moderation).

License

See LICENSE. The MCP server package is intended to be released under Apache 2.0; full platform license TBD.

About

Self-hostable MCP server + webapp that turns AI coding sessions (Claude Code, Cursor, Windsurf) into durable, semantically-retrievable skills and a grounded Oracle. Provider-agnostic (Gemini/GLM/OpenAI/Claude), production-ready on a bare VM.

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Packages

 
 
 

Contributors