Context Cortex

Context Cortex is a domain-agnostic personal research operating system: continuously ingest documents and web findings, index them in LanceDB (hybrid vector + full-text search with Voyage AI reranking), maintain a git-backed markdown memory, and work from Claude Desktop via the Model Context Protocol (MCP) or from the bundled Express server (RSS, scheduled research, HTTP ingest, legacy UI).

It evolved from a ResearchBrain-style codebase (transfer-pricing–focused upstream) with practice-specific defaults replaced by files under memory/config/.

Features at a glance

Area	What you get
Retrieval	Hybrid vector + FTS on LanceDB, optional multi-query expansion (Claude), Voyage rerank, session context, filters (topic, jurisdiction, `concept_path`, technique tags, tier, doc status, recency).
Ingestion	Raw text, PDF (pdf-parse or optional Docling), DOCX / MD / TXT, RSS with paywall scraping (Puppeteer), Perplexity (manual + scheduled tiers), OpenClaw Gmail/Telegram PDF endpoints.
Graph metadata	At ingest, Claude can fill `entity_refs`, `regulation_refs`, `tp_method_tags` (legacy column name; use as technique/facet tags), `concept_path` validated against your `taxonomy.json`.
Memory	Markdown under `memory/` (topics, briefs, articles, sources, synthesis, config); git sync to GitHub on writes; pull on MCP/Express startup and every 30 minutes while Express runs.
Curated KB	Concept articles in `knowledge-base/concepts/`, outputs in `knowledge-base/outputs/`, registry (`MASTER_INDEX`, `ACTIVITY_LOG`, `LINT_REPORT`), compile, lint, file_output loop.
MCP	26 tools + 3 resources for Claude Desktop (search, ingest, Perplexity, agenda, briefs, compiler, linter, citations, DOCX, weekly topic scoring, observability, chunk deletion).
Express	REST API for chat (orchestrator), clip/research, briefs, upload, RAG retrieve/verify/search/dashboard, authenticated ingest for OpenClaw; node-cron jobs (RSS, briefs, synthesis, maintenance, scheduled research).

Quick start

cp .env.example .env and set API keys. Use CONTEXT_CORTEX_API_KEY for Bearer-protected ingest routes; RESEARCHBRAIN_API_KEY is still accepted as an alias.
Edit memory/config/taxonomy.json, research-agenda.md, and corpus-profile.md for your domain.
Optionally fill perplexity-queries.json so scheduled tiers (current_awareness, thematic, horizon) run non-empty queries.
npm install, then npm run validate-env (requires live keys) or npm run dev to start Express on port 3000 (or PORT).
Point Claude Desktop MCP at mcp-server.js (see MCP server).

Embeddings: .env.example defaults to voyage-finance-2. For non-finance corpora, set EMBEDDING_MODEL (and matching EMBEDDING_DIMENSIONS) to a general-purpose Voyage model per Voyage AI docs.

Architecture

┌─────────────────────────────────────────────────────────────┐
│              Claude Desktop (or other MCP client)            │
│                     MCP stdio transport                      │
└──────────────────────────┬──────────────────────────────────┘
                           │
              ┌────────────▼────────────┐
              │      mcp-server.js       │
              │   26 tools · 3 resources │
              └────────────┬────────────┘
                           │
   ┌───────────┬───────────┼───────────┬───────────┐
   │           │           │           │           │
┌──▼───┐ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ ┌────▼─────┐
│ RAG  │ │ Memory  │ │Perplexity│ │   KB    │ │ Output   │
│Lance │ │  + git  │ │  Sonar   │ │compiler │ │  loop    │
└──┬───┘ └─────────┘ └──────────┘ └─────────┘ └──────────┘
   │
┌──▼──────────────────────────────────────────────────────┐
│              Express server.js (:3000)                   │
│  RSS · crons · OpenClaw ingest · RAG HTTP · dashboard     │
└──────────────────────────────────────────────────────────┘

RAG pipeline

Primary code lives under src/rag/.

Component	Role
`retriever.js`	Main retrieval entry: hybrid search + rerank; optional query expansion (`QUERY_EXPANSION_ENABLED`, `retrievalPipeline.js`). Logs retrieval events for observability.
`vectorStore.js`	LanceDB Cloud: upserts, filters, Phase 2 schema toggles (`LANCEDB_PHASE2_SCHEMA`), graph column probing.
`embedder.js`	Voyage embeddings; disk cache (`EMBEDDING_CACHE_PATH`).
`reranker.js`	Voyage `rerank-2.5` (or `VOYAGE_RERANK_MODEL`) second-stage ranking.
`chunker.js`	Sentence/chunk boundaries (`compromise` NLP), size limits and overlap for non-Docling text.
`hierarchicalChunker.js`	Docling path: section breadcrumbs, `section_path`, merge/split without overlap.
`ingestionPipeline.js`	Chunk → embed → graph metadata → upsert → markdown sidecar → git sync / registry hooks. Entry points: `ingest()`, `ingestPages()`, `ingestFromParsedDoc()`.
`graph-metadata-extractor.js`	Claude extracts structured fields; `concept_path` validated against `memory/config/taxonomy.json`.
`concept-taxonomy.js`	Loads taxonomy from disk; `reloadTaxonomy()` if you change JSON at runtime.
`validation.js`	Safe upsert validation; failed tagging can mark `status`.
`maintenance.js`	TTL expiry (`deleteExpired`), knowledge-gap analysis → scheduler hints.

PDFs: Set DOCLING_ENABLED=true and install Python deps from ingestion/python/requirements.txt for layout-aware parsing via scripts/docling_parse.py; otherwise pdf-parse. Metadata inference in src/ingest/pdfProcessor.js.

Citations: Drafts can use [CHUNK_ID: claim]; src/output/citationVerifier.js checks against LanceDB (and optionally Claude) — exposed as MCP verify_citations and POST /api/rag/verify-citations.

Memory layer

Implemented in src/memory.js.

Path	Purpose
`memory/topics/`	Topic notes (append via MCP `save_memory_note`).
`memory/articles/`	Per-ingestion markdown (RSS, uploads, Perplexity, etc.).
`memory/briefs/`	Daily intelligence briefs (`YYYY-MM-DD.md`).
`memory/sources/`	Source logs (ingested lines per feed).
`memory/synthesis/`	`master-thesis.md`, `weekly-synthesis.md`.
`memory/config/`	`research-agenda.md`, `taxonomy.json`, `corpus-profile.md`, `perplexity-queries.json`, etc.
`memory/conversations/`	Optional conversation logs when using Express chat.

Writes trigger syncToGitHub when GITHUB_TOKEN and GITHUB_REPO are set. MEMORY_PATH can point to an external volume.

Knowledge base layer

Path / module	Purpose
`knowledge-base/concepts/`	Compiled concept articles (markdown + frontmatter).
`knowledge-base/outputs/`	Filed white papers, alerts, query responses (`file_output` / output loop).
`knowledge-base/registry/`	`MASTER_INDEX.md`, `ACTIVITY_LOG.md`, `LINT_REPORT.md`, `SCHEMA_CHANGELOG.md`.
`src/knowledgeCompiler.js`	`discoverConcepts`, `compileArticle`, batch `compileAll`, staleness, schema suggestions. Uses `corpus-profile.md` then `SOUL.md` for tone.
`src/knowledgeLinter.js`	Consistency, staleness, coverage, orphans, citations, cross connections, research questions.
`src/knowledgeRegistry.js`	Registry queries and updates.
`src/outputLoop.js`	Files outputs, summary chunk, re-ingest, stale concept flags.

MCP server (`mcp-server.js`)

Server name: context-cortex
Transport: stdio (stdout must remain JSON-RPC only; logging goes to stderr)
Startup: optional git pull via src/git-sync.js; optional LanceDB Phase 2 preflight when LANCEDB_PHASE2_SCHEMA=true

Resources (3)

URI	Description
`contextcortex://system-prompt`	Session system prompt: corpus profile / SOUL, master thesis, agenda, topic files, concept index, recent ingested lines.
`contextcortex://master-thesis`	Contents of `memory/synthesis/master-thesis.md`.
`contextcortex://agenda`	Raw `memory/config/research-agenda.md`.

Tools (26)

Tool	Purpose
`search_knowledge_base`	Primary RAG search; filters: topic, jurisdiction, content_type, `concept_path_prefix`, technique tag (`tp_method` param maps to `tp_method_tags`), tier_max, doc_status, days_ago; `mode`: `standard` \| `deep_dive`; optional `session_id` for context + `rate_last_retrieval`.
`search_by_ref`	Filter-only retrieval by `entity_ref`, `regulation_ref`, technique tag, or `concept_path_prefix` (no embedding/rerank).
`rate_last_retrieval`	Save good/bad feedback for the last retrieval in a session.
`ingest_content`	Ingest arbitrary text into LanceDB + pipeline (metadata, TTL, etc.).
`ingest_file`	Ingest local PDF, DOCX, MD, TXT by path; Docling path when enabled; updates memory topic sidecars.
`search_perplexity`	Live web research via Perplexity Sonar Pro.
`get_research_agenda` / `update_research_agenda`	Read/update `memory/config/research-agenda.md` (core topics + search terms).
`get_articles_digest`	List ingested RSS-style articles (optional date/source filters).
`get_knowledge_stats`	Chunk totals, topic distribution, date range.
`get_ingestion_dashboard`	Rich dashboard: periods (24h, 48h, local “today”, week, month, year), distributions (topic, content type, channel, source, tier, doc status).
`verify_citations`	Verify `[CHUNK_ID: claim]` citations in a draft.
`get_daily_brief`	Load brief by date or `generate: true` for today (Perplexity + Claude; requires agenda core topics).
`save_memory_note`	Append structured note to `memory/topics/<file>.md`.
`get_master_thesis`	Load master thesis markdown.
`compile_concept`	Build/rebuild a concept article from chunks (`force_recompile`, `scope`: full \| incremental).
`discover_concepts`	Suggest new concept slugs from chunk density (optional `domain_filter` on taxonomy L1).
`lint_knowledge_base`	KB health report; `preset`: quick \| full; optional `checks` array; `output_format`: summary \| full_report \| actionable_only.
`file_output`	File a finished artifact into `knowledge-base/outputs/` and re-ingest summary (`output_type` enum includes white_paper, client_alert, query_response, etc.).
`knowledge_registry`	`query_type`: summary, domain_coverage, recent_additions, stale_articles, source_list, activity_log.
`convert_to_docx`	Markdown → Word via Python `python-docx` (`src/weekly-topics/convert_to_docx.py`).
`score_weekly_topics`	Deterministic selection from scored candidates (recency, impact, rigor, novelty, whitespace + diversity / repeat rules).
`get_observability_report`	JSON summary over last N days (retrieval, ingestion, MCP usage, tokens when logged).
`get_low_score_queries`	Recent queries with low reranker scores.
`get_recent_alerts`	Threshold alerts from `cc-alerts.jsonl`.
`delete_chunks`	LanceDB SQL predicate deletion; `confirm: true` required after dry-run.

Example MCP config (paths and keys are yours to fill):

{
  "mcpServers": {
    "context-cortex": {
      "command": "node",
      "args": ["/absolute/path/to/context-cortex/mcp-server.js"],
      "env": {
        "ANTHROPIC_API_KEY": "...",
        "VOYAGE_API_KEY": "...",
        "LANCEDB_URI": "...",
        "LANCEDB_API_KEY": "...",
        "PERPLEXITY_API_KEY": "...",
        "GITHUB_TOKEN": "...",
        "CONTEXT_CORTEX_API_KEY": "...",
        "MEMORY_PATH": "./memory"
      }
    }
  }
}

Note: npm run mcp does not run RSS or Express crons — use npm start / npm run dev for background automation, or run npm run ingest-rss-once manually.

Express server (`server.js`)

Default port 3000 (PORT). Selected routes:

Health and status

Method	Path	Notes
GET	`/api/health`	Liveness.
GET	`/api/status`	Broader status payload.
GET	`/api/rag/health`	RAG / LanceDB oriented check.

Chat (orchestrator)

Method	Path	Notes
GET	`/api/chat/start`	Start session.
POST	`/api/chat`	User message → Claude with memory/RAG context.
POST	`/api/chat/end`	End session.
GET	`/api/session`	Session snapshot.

Agenda, sources, clip, research, briefs

Method	Path	Notes
GET/PUT	`/api/agenda`	Read/update research agenda.
GET	`/api/sources/status`, `/api/sources/verify`	Feed / auth status.
POST	`/api/sources/authenticate`	Cookie / premium flow helpers.
POST	`/api/clip`, GET `/api/clip/detect`, POST `/api/clip/paste`	URL clipping pipeline.
POST	`/api/research/search`, `/api/research/save`	Research helpers.
GET	`/api/briefs`, `/api/briefs/:date`	List / load briefs.
POST	`/api/briefs/generate`	Trigger daily brief.

Ingestion and uploads

Method	Path	Notes
POST	`/api/ingest`	Generic ingest hook.
GET	`/api/ingest/stats`	Ingest stats.
GET	`/api/reprocess/candidates`, POST `/api/reprocess`	Reprocess flow.
GET	`/api/articles`	Article listing.
POST	`/api/upload`	Raw upload (size limit `UPLOAD_MAX_SIZE_MB`).
GET	`/api/upload/formats`	Supported MIME/extensions.
GET	`/api/debug/uploads`	Debug helper.

RAG HTTP API

Method	Path	Auth	Notes
POST	`/api/rag/ingest`	Bearer `CONTEXT_CORTEX_API_KEY` or `RESEARCHBRAIN_API_KEY`	JSON ingest for agents (e.g. OpenClaw writer).
POST	`/api/rag/ingest/pdf`	Bearer	Base64 PDF + metadata.
POST	`/api/rag/ingest/gmail-pdf`	Bearer	OpenClaw Gmail skill shape.
POST	`/api/rag/ingest/telegram-pdf`	Bearer	OpenClaw Telegram skill; optional async queue (`RAG_INGEST_QUEUE_ENABLED`).
GET	`/api/rag/stats`	—	Vector stats.
GET	`/api/rag/dashboard`	—	Ingestion dashboard JSON (timezone `DASHBOARD_TIMEZONE`, scan caps in `.env.example`).
GET	`/api/rag/search`	—	Query params for hybrid search.
POST	`/api/rag/retrieve`	—	JSON body `{ query, filters, topK, topN }` for tools like OpenClaw.
POST	`/api/rag/verify-citations`	—	JSON `{ draft_text }`; optional Telegram alert on flags.

Static UI

public/index.html — legacy web UI.
public/dashboard.html — ingestion dashboard.

Scheduled jobs (crons)

All schedules run only while the Express process is running (npm start / npm run dev).

Schedule	Job
`RSS_POLL_INTERVAL` `6h` → `0 /6 * `; any other value → hourly*	`ingestFeeds()` — RSS + gates (agenda, dedupe, age, optional `SOURCES_ENABLED`).
Daily 7:00	`generateDailyBrief()` — Perplexity sweep from agenda topics → Claude brief → `memory/briefs/`.
Sunday 8:00	Weekly synthesis appended to `memory/synthesis/weekly-synthesis.md` (prompt is in `server.js` — customize for your domain).
1st of month 4:00	Full knowledge base lint → `knowledge-base/registry/LINT_REPORT.md`.
Sunday 9:00	`compileAll()` — recompile stale concept articles.
`SCHEDULE.current_awareness` (default `0 /4 * *`)	Run next `current_awareness` query from `perplexity-queries.json` (skips if empty).
`SCHEDULE.thematic` (default Monday 2:00)	Thematic tier.
`SCHEDULE.horizon` (default 1st of month 3:00)	Horizon tier.
Daily 3:00	RAG maintenance: TTL cleanup + gap analysis (`maintenance.js`).
Monday 6:00	Court monitor — only if `COURT_MONITOR_ENABLED=true` (ingest path still uses tax-case defaults in code; adjust if you reuse this job).
Monday 7:00	OECD monitor — only if `OECD_MONITOR_ENABLED=true` (same note).
Every 30 minutes	`git pull` to sync memory from GitHub.

Weekly CLI pipeline (not Express): scripts/weekly-trigger.sh + skill skills/weekly-topics-whitepaper/SKILL.md.

Configuration

File	Role
`memory/config/README.md`	Index of config files.
`memory/config/taxonomy.json`	Allowed `concept_path` L1/L2 prefixes for graph metadata + discovery.
`memory/config/research-agenda.md`	Core topics + search terms for RSS gating and daily brief Perplexity queries.
`memory/config/corpus-profile.md`	Voice, audience, citation norms for compiler and linter (preferred over `SOUL.md`).
`memory/config/perplexity-queries.json`	Arrays: `current_awareness`, `thematic`, `horizon` — each item `{ id, query, metadata }`.
`memory/config/perplexity-queries.example.json`	Copy/paste starter shape.
`src/config/sources.js`	RSS feed rows + env-built premium feeds.
`src/config/premiumRssUrls.js`	Default URL lists (empty by default; set env vars to enable).

LanceDB / taxonomy: If you point at an existing table built under another taxonomy, old concept_path values will not automatically match a new taxonomy.json. Prefer a new table/project, or run scripts/backfill-graph-metadata.js after changing taxonomy.

Environment variables

Copy .env.example to .env. Highlights:

Category	Variables (non-exhaustive)
LLM / search	`ANTHROPIC_API_KEY`, `PERPLEXITY_API_KEY`, optional Perplexity Agent flags.
Embeddings / rerank	`VOYAGE_API_KEY`, `EMBEDDING_MODEL`, `EMBEDDING_DIMENSIONS`, `VOYAGE_RERANK_MODEL`, batching / delay.
LanceDB	`LANCEDB_URI`, `LANCEDB_API_KEY`, `LANCEDB_PHASE2_SCHEMA`, `LANCEDB_UPSERT_BATCH_SIZE`, `PHASE2_PREFLIGHT_STRICT`.
Auth	`CONTEXT_CORTEX_API_KEY` (preferred), `RESEARCHBRAIN_API_KEY` (legacy).
Git	`GITHUB_TOKEN`, `GITHUB_REPO`, `GITHUB_BRANCH`.
Paths	`MEMORY_PATH`, `COOKIE_DIR`, `EMBEDDING_CACHE_PATH`.
RSS	`RSS_POLL_INTERVAL`, `SOURCES_ENABLED`, `RSS_MAX_PER_FEED`, `RSS_MAX_AGE_DAYS`, publisher URL env vars (see `.env.example`).
PDF	`DOCLING_ENABLED`, `UPLOAD_MAX_SIZE_MB`.
Retrieval	`QUERY_EXPANSION_ENABLED`, `HYDE_ENABLED`, `DEEP_DIVE_MAX_CHUNKS`, graph flags.
OpenClaw / skills	`GMAIL_PDF_SKILL_ENABLED`, `TELEGRAM_PDF_SKILL_ENABLED`, `MY_EMAIL`, `TRUSTED_GMAIL_SENDERS`, `MY_TELEGRAM_USER_ID`, queue flag.
Dashboard	`DASHBOARD_TIMEZONE`, `DASHBOARD_MAX_SCAN_ROWS`, etc.
Optional monitors	`COURT_MONITOR_ENABLED`, `OECD_MONITOR_ENABLED`.

npm scripts

Script	Command	Purpose
`start`	`node server.js`	Production Express.
`dev`	`node --watch server.js`	Dev server with reload.
`mcp`	`node mcp-server.js`	MCP stdio only.
`validate-env`	`node scripts/validateEnv.js`	Keys + API smoke tests.
`validate-sources`	`node scripts/validate-sources.js`	HTTP check all RSS URLs.
`validate-premium-feeds`	`node scripts/validate-premium-feeds.js`	Economist/WSJ URL lists.
`ingest-rss-once`	`node scripts/ingest-rss-once.js`	One-shot RSS pass.
`migrate-memory`	`node scripts/migrateMemory.js`	Bulk index `memory/articles` → LanceDB.
`test-rag`	`node scripts/testRag.js`	RAG regression tests (needs LanceDB + Voyage env).
`test-mcp`	`node scripts/testMcp.js`	MCP registry + tool smoke (needs full env).
`schema-guard`	compound	Phase 2 columns + graph preflight + Lance check + `test-rag`.
`import-cookies`	`node scripts/import-cookies.js`	Premium feed cookies.

OpenClaw and external ingest

Skill YAMLs: openclaw/skills/ (gmail-pdf-ingest, telegram-pdf-ingest, context-cortex-writer, retrieval-feedback).
Install symlinks: ./scripts/install-openclaw-skills.sh
Set CONTEXT_CORTEX_API_KEY (or legacy key) and your Express base URL in OpenClaw.

Observability

JSONL logs (gitignored): logs/cc-observability.jsonl, logs/cc-alerts.jsonl
MCP tools: get_observability_report, get_low_score_queries, get_recent_alerts
Optional: scripts/dead-chunk-detector.js (compare Lance doc_ids to retrieval logs)

Deployment and ops

Docker: Dockerfile — Node 20 + Chromium for Puppeteer.
Railway: examples/railway.json — health GET /api/health.
macOS launchd: launchd/com.contextcortex.*.plist + scripts/setup-launchd.sh.
CI: .github/workflows/schema-drift-guard.yml — requires CONTEXT_CORTEX_API_KEY or RESEARCHBRAIN_API_KEY secret among others.

Full Mac mini / volume notes: README-deployment.md.

License

See Polyform Noncommerical 1.0.0. (LICENSE).

Document	Contents
README-workflow.md	Day-to-day usage, MCP patterns, weekly topics pipeline.
README-deployment.md	launchd, logs, OpenClaw on a server, Railway env checklist.
CLAUDE.md	Repo map for Claude Code / contributors.
memory/config/README.md	First-edit checklist for new domains.

Navigation Menu

Repository files navigation

Context Cortex

Table of contents

Features at a glance

Quick start

Architecture

RAG pipeline

Memory layer

Knowledge base layer

MCP server (`mcp-server.js`)

Resources (3)

Tools (26)

Express server (`server.js`)

Health and status

Chat (orchestrator)

Agenda, sources, clip, research, briefs

Ingestion and uploads

RAG HTTP API

Static UI

Scheduled jobs (crons)

Configuration

Environment variables

npm scripts

OpenClaw and external ingest

Observability

Deployment and ops

Further reading

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
.github/workflows		.github/workflows
bot		bot
examples		examples
ingestion/python		ingestion/python
knowledge-base/registry		knowledge-base/registry
launchd		launchd
memory		memory
openclaw/skills		openclaw/skills
public		public
schemas		schemas
scripts		scripts
skills		skills
src		src
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README-deployment.md		README-deployment.md
README-workflow.md		README-workflow.md
README.md		README.md
SOUL.md		SOUL.md
mcp-server.js		mcp-server.js
observability.js		observability.js
package-lock.json		package-lock.json
package.json		package.json
server.js		server.js

Folders and files

Latest commit

History

Repository files navigation

Context Cortex

Table of contents

Features at a glance

Quick start

Architecture

RAG pipeline

Memory layer

Knowledge base layer

MCP server (mcp-server.js)

Resources (3)

Tools (26)

Express server (server.js)

Health and status

Chat (orchestrator)

Agenda, sources, clip, research, briefs

Ingestion and uploads

RAG HTTP API

Static UI

Scheduled jobs (crons)

Configuration

Environment variables

npm scripts

OpenClaw and external ingest

Observability

Deployment and ops

Further reading

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

MCP server (`mcp-server.js`)

Express server (`server.js`)