Skip to content

feat: Open Brain v1.0 -- unified semantic knowledge backend for PAI - #1

Merged
rodaddy merged 62 commits into
mainfrom
wip/open-brain-init
Mar 15, 2026
Merged

feat: Open Brain v1.0 -- unified semantic knowledge backend for PAI#1
rodaddy merged 62 commits into
mainfrom
wip/open-brain-init

Conversation

@rodaddy

@rodaddy rodaddy commented Mar 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Complete MCP server with PostgreSQL + pgvector semantic search (6 tools, 5 tables, 768-dim embeddings)
  • Phase 6 PAI integration: hooks, skills, and legacy KB data migrated to Open Brain
  • Session-load hook replaces 3 old hooks with unified brain context + session injection
  • Session-save hook captures git context (commits, branch) before compaction
  • 318 legacy KB entries imported (decisions, learnings, patterns) with 2,445 embeddings
  • /brain and /capture-session skills rewritten for Open Brain
  • Per-session MCP server instances fix concurrent client support

What's Included

Core (Phases 1-5):

  • PostgreSQL schema (5 tables + pgvector HNSW indexes)
  • MCP server with StreamableHTTPServerTransport + Bearer auth (5 roles)
  • 6 tools: log_thought, log_decision, search_brain, find_person, session_save, session_load
  • Embedding service via LiteLLM (gemini-embedding-001)
  • 170 tests, CI pipeline, deployment template

Integration (Phase 6):

  • Enhanced session-load hook (tag detection + search_brain + session_load)
  • Enhanced session-save hook (git commits as key_decisions)
  • Legacy KB import script with noise filtering + content dedup
  • Settings.json updated (old hooks removed)
  • OPEN_BRAIN_AGENT_TOKEN exported in shell profile

Test plan

  • 170 unit tests pass
  • Typecheck clean
  • Session-load with/without package.json
  • Session-load/save graceful degradation (no token)
  • Session-save on git and non-git directories
  • 3 concurrent MCP sessions (transport fix)
  • mcp2cli search_brain, log_thought, find_person
  • Semantic search returns relevant results (distance 0.17-0.25)

🤖 Generated with Claude Code

rodaddy and others added 30 commits March 13, 2026 14:40
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 plans in 3 waves, all 5 requirements covered (DB-01, DB-02, SRV-01, AUTH-01, DATA-02).
Revision added migration integration test for DB schema verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- package.json with all Phase 1 dependencies (MCP SDK, pg, pgvector, express, zod, cors)
- tsconfig.json targeting ESNext with strict mode
- bunfig.toml with 80% coverage thresholds
- .env.example documenting all 11 required environment variables
- src/types.ts exporting Role, Table, Permission, AuthInfo, PoolHealth, HealthStatus
- src/logger.ts structured JSON logger with info/warn/error methods

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- src/db/pool.ts with createPool (pgvector type registration per connection) and checkPoolHealth
- src/db/migrations/001_init.sql with all 5 tables, halfvec(768) columns, HNSW indexes using halfvec_cosine_ops
- src/db/migrate.ts with runMigrations tracking applied migrations in _migrations table
- scripts/migrate.ts CLI entry point for running migrations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add 001_init.test.ts with 12 assertions proving schema correctness
- Fix graceful skip: use top-level await instead of beforeAll for DB check
- Fix _migrations table collision: add IF NOT EXISTS in SQL
- Fix atttypmod assertion: pgvector 0.8.1 stores raw dimension (768)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Full 5-role x 5-table matrix coverage in permissions.test.ts
- Middleware unit tests for Bearer token validation in auth.test.ts
- buildTokenMap and verifyToken unit tests
- Stub implementations for type-checking (44 of 64 tests fail)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- LiteLLM embedding client with 5s AbortController timeout
- Graceful null return on any failure (non-200, network, timeout)
- SHA-256 content hashing with whitespace normalization
- TDD: full coverage with mocked fetch in embedding.test.ts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bearer token validation with constant-time comparison
- Role-permission matrix for 5 roles x 5 tables
- TDD: full matrix coverage in permissions.test.ts
- TDD: middleware unit tests in auth.test.ts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- McpServer factory with StreamableHTTPServerTransport sessions
- Transport handlers for POST/GET/DELETE /mcp with session management
- Express app with cors, health endpoint, auth-gated MCP routes
- Testable createApp factory with dependency injection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Health endpoint: 200 healthy, 503 degraded, no auth required
- Auth rejection: 401 for missing and invalid tokens
- MCP initialize: valid token creates session with session ID

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Catch async MCP handler errors to prevent process crash
- Add session TTL (30min) and max sessions cap (100) to transport
- Remove ALTER DATABASE from migration, use pool-level statement_timeout
- Add graceful shutdown (SIGTERM/SIGINT)
- Add updated_at auto-update triggers for 4 tables
- Store auth identity per MCP session, verify on subsequent requests
- Restrict CORS to ALLOWED_ORIGINS env var
- Fail fast on missing DB_HOST, DB_USER env vars
- Add 1MB body size limit to express.json
- Catch pgvector registerTypes failures
- Fatal exit if zero auth tokens configured
- Validate embedding input length (max 32KB)
- Freeze permission Sets to prevent mutation
- Remove unused uuid-ossp extension
- Isolate test mock mutations with beforeEach/afterEach

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- log_thought: insert to thoughts with embedding, content_hash dedup, graceful degradation
- log_decision: insert to decisions with permission enforcement, title+rationale embedding
- Tool orchestrator (registerAllTools) wires both tools into McpServer
- src/index.ts wired: registerAllTools called between createBrainServer and createTransportHandlers
- 16 tests: unit tests for both tools + InMemoryTransport protocol tests
- All 108 project tests pass, TypeScript compiles cleanly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…c search

- Cross-table CTE UNION ALL with cosine distance ranking across all 5 tables
- Permission-filtered table access (canRead per table, discord blocked entirely)
- Optional table filter restricts search to single table
- Embedding failure returns isError (search requires query vector)
- Dynamic SQL construction enforces permissions at query build time
- 12 unit tests + 3 protocol tests, 100% coverage on search-brain.ts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Name mode: ILIKE partial match with special char escaping (%, _)
- Semantic mode: cosine distance ranking via pgvector halfvec
- Default mode falls back to name search when omitted
- Permission check: discord denied (NONE on relationships)
- No-results returns informational message (not isError)
- Limit defaults to 5, configurable up to 20
- 9 unit tests + 2 protocol tests, all 134 project tests green

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rodaddy and others added 20 commits March 13, 2026 17:53
- SUMMARY.md with 2 tasks, 3 auto-fixed deviations documented
- STATE.md updated: phase 4 plan 3/3, 70% progress
- ROADMAP.md updated: phase 04 marked complete (3/3 plans)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SUMMARY.md with task commits, decisions, and deviations
- STATE.md updated with progress (80%), decisions, session info
- ROADMAP.md updated with phase 04 plan progress

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Retrieves existing tokens from vaultwarden (not generate new ones)
- Supports --verify mode to check all 5 role tokens are present
- Supports --rotate mode for future token rotation
- All 5 tokens confirmed present: admin, agent, discord, n8n, readonly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…e.md

- Added open-brain HTTP backend entry to ~/.config/mcp2cli/services.json
- Uses AUTH_TOKEN_AGENT from vaultwarden for Bearer auth
- Created .planning/agent-reference.md with endpoints, tools, and examples
- Server not yet deployed -- mcp2cli config verified via services listing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SUMMARY.md with token verification, mcp2cli registration, agent-reference.md
- STATE.md updated to phase 5, plan 1 of 2, 90% progress
- ROADMAP.md updated with phase 4 complete, phase 5 plan 01 complete
- REQUIREMENTS.md INT-01 marked complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- PreCompact hook calls session_save on Open Brain before context compaction
- SessionStart hook calls session_load and outputs formatted context to stdout
- Both hooks handle missing tokens and server errors gracefully (silent exit 0)
- Updated global ~/.claude/settings.json with PreCompact and SessionStart entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Built 6-node n8n workflow: webhook -> extract -> MCP init -> session ID -> log_thought -> respond
- Workflow handles two-step MCP handshake (initialize + tools/call with session header)
- Captures Discord messages as thoughts with username, channel tags
- Workflow ID: n3BDmv0iqbG470wy, active at /webhook/open-brain-thought
- Updated agent-reference.md with consumer integration details

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…plan

- SUMMARY.md with self-check passed (9/9 artifacts verified)
- STATE.md updated: 100% complete, all 10 plans across 5 phases done
- ROADMAP.md phase 5 marked complete
- REQUIREMENTS.md INT-02 marked complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LiteLLM proxy requires Bearer token authentication. Without the header,
embedding requests return 401 and all embeddings silently degrade to NULL.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LiteLLM proxy requires Bearer auth for embedding requests.
Added to .env.schema, .env.example, and fetch-secrets.sh.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v1.0 complete and deployed. Phase 6 carries forward the migration
analysis: 5 knowledge stores mapped, 4 injection hooks analyzed,
3 capture pipelines documented, all file paths and decisions locked.

Pre-req: install skippy-agentspace before planning.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enhanced session-load hook to absorb inject-brain-context and query-knowledge
functionality -- one hook replaces three, with tag detection from package.json
and parallel search_brain + session_load calls.

Enhanced session-save hook to extract git commits as key_decisions and branch
context for richer pre-compaction summaries.

Added legacy KB import script (decisions, learnings, patterns) with noise
filtering, content dedup via content_hash, and safe date handling.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each initialize request now creates a fresh McpServer to avoid
"Already connected to a transport" errors with concurrent clients.
Fixed hook endpoint from database host (10.71.20.49) to actual
server (10.71.20.15). Added required Accept header for MCP SDK.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The MCP SDK returns SSE format (event/data lines) not plain JSON.
Added SSE response parser to session-load hook and Accept headers
to both hooks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant