MCP Vector Server

Note: This is a private, local-only project. Public deployment, PRs, and CI/CD are out of scope.

A production-ready Model Context Protocol (MCP) server that enables AI assistants (like ChatGPT) to perform semantic search over local project documentation and codebases. The server ingests documents from sandbox/, stores text chunks and embeddings in PostgreSQL with the pgvector extension, and exposes MCP tools for intelligent knowledge retrieval.

---

Purpose

• Local Knowledge Management: Keep project context close to development by embedding local files with structured metadata tags like [persona:dev] or [project:example1].
• Semantic Search: Enable AI assistants to query documentation and code using natural language, powered by OpenAI embeddings and vector similarity search.
• Multi-Tenant Architecture: Isolate projects using dedicated PostgreSQL databases (mcp_vector_<project>), ensuring data separation and scalable performance.
• MCP Protocol: Expose standardized tools via the Model Context Protocol, making the server compatible with ChatGPT and other MCP-compliant clients.

---

Features

Current (v0.1.0)

✅ MCP Server (TypeScript + @modelcontextprotocol/sdk)

• HTTP/SSE transport for remote access via Cloudflare tunnel
• stdio transport for local development
• Six production-ready tools: db.list, search, fetch, read, write, ingest.changed, sandbox.read
• Two resources: corpus://info, status://health

✅ Vector Store (PostgreSQL 16+ with pgvector)

• Database-per-project multi-tenancy (mcp_vector_example1, etc.)
• IVFFlat vector indexing for fast approximate nearest neighbor search
• GIN indexes on JSONB metadata for efficient filtering
• Connection pooling with graceful shutdown

✅ Ingestion Pipeline

• Multi-granular chunking: document, recursive (~700 tokens), code-aware
• Metadata extraction from bracketed tags: [persona:dev], [project:example1]
• OpenAI embeddings API integration (text-embedding-3-small, 1536 dimensions)
• Incremental updates: upserts changed files, removes deleted ones

✅ Testing & Quality

• 290 tests (270 unit + 20 integration)
• 80.54% statement coverage, 83.55% function coverage
• Comprehensive error handling with VectorStoreError
• <5% code duplication (jscpd verified)

---

Prerequisites

Requirement	Version	Purpose
Node.js	20.x LTS	Runtime environment
npm	10.x+	Package manager
PostgreSQL	16.x	Database server
pgvector	0.7.0+	Vector similarity extension
OpenAI API Key	-	Embedding generation

Verify versions:

node --version    # v20.x.x
npm --version     # 10.x.x
psql --version    # PostgreSQL 16.x

---

Quick Start

1. Clone and Install

git clone https://github.com/your-org/mcp-vector-server.git
cd mcp-vector-server
npm ci

2. Start PostgreSQL

Option B: Local Installation (Recommended for this setup)

brew services start postgresql@16  # macOS
# or
sudo systemctl start postgresql     # Linux

Optional (Docker Compose): if you prefer containers, you can start a Postgres service defined in docker-compose.yml.

docker compose up -d postgres

3. Configure Environment

Create .env file:

# Database
DATABASE_URL=postgres://mcp:mcp@127.0.0.1:5432/mcp_vector_default

# OpenAI
OPENAI_API_KEY=sk-...your-api-key...
OPENAI_EMBEDDING_MODEL=text-embedding-3-small

# Server (optional)
MCP_HTTP_HOST=127.0.0.1
MCP_HTTP_PORT=3333
MCP_HTTP_PATH=/mcp

4. Initialize Database

# Run migrations
npm run db:migrate

# Create project databases
npm run seed:projects

5. Ingest Sample Data

# Ingest all projects from sandbox/
npm run ingest

# Or ingest specific project
npm run ingest -- --root-dir ./sandbox/example1

6. Start the Server

# Development mode (auto-restart on changes)
npm run dev

# Production mode (compiled)
npm run build
npm start

Expected output:

MCP Vector Server starting...
✓ Database connection verified
✓ OpenAI API key configured
✓ Loaded 2 projects: example1, example2
Server listening on http://127.0.0.1:3333/mcp

7. Test the Server

# List available databases
npm run tool:run -- db.list

# Run a search query
npm run tool:run -- search --query "authentication setup" --databaseId example1

# Read a file directly (sandbox)
npm run tool:run -- sandbox.read --path "docs/README.md" --databaseId example1

# Create or modify repo files safely (non-sandbox)
npm run tool:run -- read path=src/server/createMcpServer.ts
npm run tool:run -- write path=docs/new-notes.md mode=create content="# Notes\nHello!\n"
# Append via patch
npm run tool:run -- write path=docs/new-notes.md mode=patch patch='{"type":"append","data":"More\n"}'

# Incremental ingestion of changed files
# CLI
npm run ingest:changed -- --database mcp_vector_example1 --root-dir ../example1 --since main
# Tool via CLI helper
npm run tool:run -- ingest.changed databaseId=example1 since=main
# Read a repo file (non-sandbox)
npm run tool:run -- read path=src/server/createMcpServer.ts

---

Available Scripts

Script	Purpose
Development
npm run dev	Start server in watch mode (tsx)
npm run build	Compile TypeScript to dist/
npm start	Run compiled server
Database
npm run db:migrate	Apply Drizzle migrations
npm run seed:projects	Create project databases from sandbox/
npm run ingest	Run ingestion pipeline
Testing
npm test	Run Jest test suites
npm test -- --coverage	Generate coverage report
npm test -- --watch	Run tests in watch mode
Code Quality
npm run lint	Check code with ESLint
npm run format	Check formatting with Prettier
npm run format:fix	Fix formatting issues
Operations
npm run tool:run	Execute MCP tools locally
npm run server:restart	Restart server (production)
npm run stack:start	Start Cloudflare tunnel
npm run stack:logs	View tunnel logs
npm run reset:stack	Clean restart (drop DBs, re-ingest)

---

Project Structure

mcp-vector-server/
├── server.ts                     # HTTP server entry point
├── src/
│   ├── config/                   # Configuration (constants, defaults)
│   ├── db/                       # Database layer
│   │   ├── pool.ts              # Connection pooling
│   │   ├── schema.ts            # Drizzle ORM schema
│   │   ├── embeddingsRepository.ts  # Data access layer
│   │   └── projectRegistry.ts   # Multi-tenant project config
│   ├── ingestion/               # Ingestion pipeline
│   │   ├── walker.ts            # File discovery
│   │   ├── chunker.ts           # Text chunking
│   │   ├── embedder.ts          # OpenAI embedding generation
│   │   ├── fileProcessor.ts     # Content reading, tag extraction
│   │   ├── metadata.ts          # Metadata parsing
│   │   ├── pathResolver.ts      # Path normalization
│   │   ├── ingest.ts            # Orchestration
│   │   └── runIngestion.ts      # CLI entry point
│   ├── server/                  # MCP server implementation
│   │   ├── createMcpServer.ts   # Server initialization
│   │   ├── configHelpers.ts     # Config parsing
│   │   ├── httpHelpers.ts       # HTTP utilities
│   │   ├── jsonRpcHelpers.ts    # JSON-RPC logging
│   │   ├── schemas.ts           # Zod validation
│   │   ├── sanitizers.ts        # Input sanitization
│   │   ├── embeddingHelpers.ts  # OpenAI client wrappers
│   │   ├── tools/               # MCP tool implementations
│   │   │   ├── dbListTool.ts
│   │   │   ├── searchTool.ts
│   │   │   ├── fetchTool.ts
│   │   │   └── sandboxReadTool.ts
│   │   └── resources/           # MCP resource implementations
│   │       ├── corpusResource.ts
│   │       └── statusResource.ts
│   ├── types/                   # Shared TypeScript types
│   └── testUtils/               # Test utilities and mocks
├── scripts/                     # Operational scripts
│   ├── seedProjects.ts
│   ├── runMigrations.ts
│   ├── runTool.ts
│   ├── startStack.ts
│   └── resetStack.ts
├── docs/                        # Documentation
│   ├── architecture.md          # System architecture
│   ├── contributing.md          # Contribution guide
│   ├── testing.md              # Testing guidelines
│   ├── error-handling.md       # Error patterns
│   └── code-smell-analysis.md  # Code quality report
├── sandbox/                     # Project files for ingestion
│   ├── example1/
│   └── example2/
├── cloudflared/                 # Cloudflare tunnel config
│   ├── config.mcp.yml
│   └── credentials/
└── db/                         # Database initialization
    └── init/
        └── 001-create-extension.sql

---

MCP Tools

db.list

Enumerate available project databases.

Input: None

Output:

{
  "databases": [
    {
      "id": "example1",
      "database_name": "mcp_vector_example1",
      "description": "Example project",
      "file_count": 42,
      "root_path": "/path/to/sandbox/example1"
    }
  ]
}

---

search

Semantic vector search across project embeddings.

Input:

{
  "query": "How do I configure authentication?",
  "databaseId": "example1",
  "top_k": 10, // Optional, default 10, max 50
  "persona": "dev", // Optional filter
  "project": "example1" // Optional filter
}

Output:

{
  "results": [
    {
      "file_path": "docs/authentication.md",
      "chunk_text": "Authentication is configured via...",
      "score": 0.87,
      "metadata": {
        "persona": ["dev", "ops"],
        "project": ["example1"],
        "granularity": "recursive"
      }
    }
  ]
}

---

fetch

Retrieve full chunk content by chunk ID.

Input:

{
  "chunkId": "docs/auth.md::0::recursive",
  "databaseId": "example1"
}

Output:

{
  "chunk_id": "docs/auth.md::0::recursive",
  "file_path": "docs/auth.md",
  "chunk_text": "# Authentication\n\n...",
  "metadata": {...}
}

---

sandbox.read

Direct file system read from project sandbox.

Input:

{
  "path": "docs/README.md",
  "databaseId": "example1"
}

Output:

{
  "content": "# Project Documentation\n\n...",
  "size_bytes": 1024
}

Security:

• Path traversal protection (../ blocked)
• Sandboxed to project root only
• 10 MB size limit

---

MCP Resources

corpus://info

Project metadata and statistics.

Output:

{
  "projects": [
    {
      "id": "example1",
      "name": "Example Project",
      "root_path": "/sandbox/example1",
      "file_count": 42,
      "total_chunks": 387
    }
  ]
}

---

status://health

Server health and configuration.

Output:

{
  "status": "healthy",
  "server_version": "0.1.0",
  "database_connected": true,
  "openai_configured": true,
  "projects_loaded": 2
}

---

Deployment

Local Development

# Start PostgreSQL
docker-compose up -d postgres

# Start server
npm run dev

# Server listens on http://127.0.0.1:3333/mcp

---

Production (with Cloudflare Tunnel)

Required if you want hosted ChatGPT to access your local MCP server. Optional for local-only usage.

1. Set up tunnel credentials:

# Login to Cloudflare
cloudflared tunnel login

# Create tunnel
cloudflared tunnel create mcp-vector

# Copy credentials to project
cp ~/.cloudflared/<tunnel-id>.json cloudflared/credentials/mcp-vector.json
cp ~/.cloudflared/cert.pem cloudflared/cert.pem

2. Configure tunnel:

Edit cloudflared/config.mcp.yml:

tunnel: <your-tunnel-id>
credentials-file: ./credentials/mcp-vector.json

ingress:
  - hostname: mcp-vector.your-domain.com
    service: http://host.docker.internal:3333
  - service: http_status:404

3. Start server + tunnel:

# Start PostgreSQL
docker-compose up -d postgres

# Start MCP server
npm start

# Start tunnel (in separate terminal)
npm run stack:start:follow

4. Test remote access:

curl https://mcp-vector.your-domain.com/mcp \
  -X OPTIONS \
  -H "Accept: application/json"

---

Configuration Reference

Environment Variables

Required:

Variable	Description	Example
DATABASE_URL	PostgreSQL connection string	postgres://user:pass@localhost:5432/dbname
OPENAI_API_KEY	OpenAI API key	sk-...
OPENAI_EMBEDDING_MODEL	Embedding model	text-embedding-3-small

Optional:

Variable	Default	Description
MCP_HTTP_HOST	127.0.0.1	Server bind address
MCP_HTTP_PORT	3333	Server port
MCP_HTTP_PATH	/mcp	Base path for MCP endpoint
MCP_HTTP_DNS_REBINDING_PROTECTION	false	Enable DNS rebinding protection
MCP_HTTP_ALLOWED_HOSTS	-	Comma-separated allowed hosts
MCP_HTTP_ALLOWED_ORIGINS	-	Comma-separated allowed origins
DEBUG_MCP_HTTP	false	Enable debug logging
IVFFLAT_PROBES	10	Number of IVFFlat probes (higher = more accurate, slower). Range: 1–1000

Note: The default base path /mcp matches the normalizePath default used by the HTTP server (see src/server/configHelpers.ts).

Database Configuration

The server uses a database-per-project multi-tenancy model:

PostgreSQL Instance
├── mcp_vector_default     # Default database (from DATABASE_URL)
├── mcp_vector_example1    # Project database (auto-created)
└── mcp_vector_example2    # Project database (auto-created)

Each project database contains:

• embeddings table with vector column
• IVFFlat index for vector similarity
• GIN index for metadata filtering

---

Chunking Strategy

The ingestion pipeline uses multi-granular chunking:

1. Document-Level

• Purpose: High-level summaries (entire file)
• Size: Up to 8000 characters
• Granularity: document
• Use Case: Quick overviews

2. Recursive Chunking

• Purpose: General semantic search
• Size: ~700 tokens with 15-20% overlap
• Granularity: recursive
• Use Case: Balanced recall vs. precision

3. Code-Aware Chunking

• Purpose: Function/class-level precision
• Size: Variable (one function/class per chunk)
• Granularity: code
• Use Case: Code search and navigation

Metadata Extraction:

Files can include bracketed tags:

[persona:dev] [project:example1]
This content is tagged for the dev persona and example1 project.

Tags are extracted and stored in the metadata JSONB column for filtering.

---

Testing

Run Tests

# All tests
npm test

# With coverage
npm test -- --coverage

# Watch mode
npm test -- --watch

# Specific file
npm test -- searchTool.test.ts

Coverage Thresholds

Metric	Current	Target
Statements	80.54%	85%
Branches	66.8%	70%
Functions	83.55%	85%

Database Integration Tests

Integration tests require a running PostgreSQL instance:

export DATABASE_URL=postgres://mcp:mcp@127.0.0.1:5432/mcp_vector_test
npm test -- --coverage

See docs/testing.md for detailed guidelines.

---

Development Guidelines

See docs/contributing.md for:

• Development environment setup
• Coding standards (from AGENTS.md)
• Git workflow and commit conventions
• Pull request process
• How to add new MCP tools
• Troubleshooting common issues

Quick Reference:

• Language: TypeScript (strict mode)
• Style: One responsibility per file, semantic naming
• Testing: TDD with Jest, ≥85% coverage target
• Formatting: Prettier + ESLint
• Commits: Conventional commits (feat:, fix:, etc.)

---

Documentation

Document	Purpose
README.md	This file - project overview and setup
plan.md	Development roadmap and task checklist
AGENTS.md	Coding standards for AI agents
docs/architecture.md	System architecture and data flows
docs/contributing.md	Contribution guide
docs/testing.md	Testing guidelines
docs/error-handling.md	Error handling patterns
docs/code-smell-analysis.md	Code quality report

---

Troubleshooting

"Cannot connect to PostgreSQL"

# Check if PostgreSQL is running
docker-compose ps postgres

# Check connection
psql $DATABASE_URL -c "SELECT 1"

# Verify credentials in .env

"OpenAI API rate limit exceeded"

# Check rate limits in OpenAI dashboard
# Reduce batch size in ingestion:
npm run ingest -- --embedding-batch-size 50

"Port 3333 already in use"

# Find process using port
lsof -i :3333

# Kill process
kill -9 <PID>

# Or use different port
MCP_HTTP_PORT=3334 npm run dev

"No results from search"

# Verify embeddings exist
npm run tool:run -- db.list

# Re-ingest if needed
npm run ingest

# Check query embedding generation
DEBUG_MCP_HTTP=true npm run dev

---

Version History

v0.1.0 (Current)

Status: Production-ready (85% complete)

Features:

• ✅ MCP server with HTTP/SSE and stdio transports
• ✅ Four production tools: db.list, search, fetch, sandbox.read
• ✅ Two resources: corpus://info, status://health
• ✅ Multi-granular chunking (document, recursive, code-aware)
• ✅ Database-per-project multi-tenancy
• ✅ IVFFlat vector indexing with metadata filtering
• ✅ Comprehensive testing (290 tests, 80.54% coverage)
• ✅ Error handling with VectorStoreError
• ✅ Code quality: <5% duplication, zero code smells

In Progress:

• 🔄 Increasing test coverage to 85%
• 🔄 Performance optimizations (caching, query tuning)

Planned:

• 📋 OAuth2.1 authentication for HTTP transport
• 📋 Redis caching layer for frequent queries
• 📋 Advanced chunking strategies (sliding window, semantic boundaries)
• 📋 OCR/ASR support for PDFs and audio

---

Version Checkpoint (October 12, 2025)

Dependencies verified:

• ✅ Node.js: 20.x LTS (Active LTS until 2026-04-30)
• ✅ PostgreSQL: 16.x (Latest stable)
• ✅ pgvector: 0.7.0+ (Latest: 0.7.4, released August 5, 2024)
• ✅ OpenAI API: text-embedding-3-small (1536 dimensions)
• ✅ MCP SDK: @modelcontextprotocol/sdk ^1.19.1

Next checkpoint: Before v1.0 release (verify upstream dependencies)

---

Future Enhancements

Planned Features

• Authentication: OAuth2.1 support for HTTP transport
• Caching: Redis layer for embedding and query caching
• Advanced Chunking: Semantic boundary detection, sliding window
• Rich Media: OCR for PDFs, ASR for audio transcription
• Monitoring: Prometheus metrics, structured logging
• UI: Web dashboard for project management and search
• Performance: Query optimization, connection pool tuning

Research Areas

• Hybrid search (vector + keyword)
• Relevance feedback and re-ranking
• Multi-modal embeddings (code + documentation)
• Real-time ingestion (file watchers)

---

Notes

• Private, local-only project. External contributions/PRs/issue trackers are not used.

---

License

MIT License - see LICENSE for details

---

Acknowledgments

• Model Context Protocol - MCP specification
• OpenAI - Embeddings API
• pgvector - PostgreSQL vector extension
• Drizzle ORM - TypeScript ORM

---

Support

• Documentation lives in the local docs/ directory.

---

Built with ❤️ using TypeScript, PostgreSQL, and OpenAI

Planned Implementation

1. MCP Server (TypeScript + @modelcontextprotocol/sdk)
   • Communicates over stdio for local development; ready to serve over HTTPS when deployed behind a connector tunnel (e.g., Cloudflare Tunnel).
   • Ships a server.ts entrypoint executed via tsx/ts-node in development and compiled for production deployment.
   • Registers tools: db.list (enumerate project databases), search (plain-text semantic lookup), fetch (retrieve chunk content), and sandbox.read (direct file reads per project).
   • Surfaces resources/prompts that help ChatGPT choose personas and explain tool usage.
2. Vector Store (PostgreSQL + pgvector)
   • Maintains a dedicated database per sandbox project (mcp_vector_<project> by default) seeded via npm run seed:projects while your local Postgres service is running.
   • Each database exposes an embeddings table holding id, file_path, chunk_text, embedding (vector), and metadata (JSONB for tag arrays, timestamps, persona/project labels).
   • IVFFlat vector index plus GIN indexes on metadata for efficient filtering by persona/project.
   • Session-level ivfflat.probes is set from IVFFLAT_PROBES (default 10) so ANN searches return results even with small datasets; the search tool can override the probe count when deeper recall is required.
3. Ingestion Pipeline
   • Walks sandbox/, filters out binaries, parses bracketed metadata, and applies multi-granular chunking:
     • Document-level summaries for quick overviews.
     • Recursive ~700-token chunks with 15–20% overlap for general semantic search.
     • Format-aware slices (function-level for code, heading-level for markdown) when applicable.
   • Calls the OpenAI embeddings API for each chunk tier, tagging rows with granularity, persona/project labels, and span metadata before upserting into PostgreSQL.
   • Reconciles deletions by clearing or soft-deleting embeddings for files that disappear between ingestion runs.
   • Exposed via the standalone CLI (npm run ingest); MCP surface is read-only for now.
4. Connector Workflow
   • Designed to register with ChatGPT’s Developer-mode connectors so the assistant can orchestrate persona-aware workflows through the MCP tools.

Prerequisites

• Node.js 20.x LTS with TypeScript tooling (tsx, ts-node) available.
• PostgreSQL 16.x with the pgvector extension 0.7.x or newer available locally; adjust image tags if upstream releases move forward.
• OpenAI API key with access to the embedding and GPT endpoints you configure (defaults target gpt-4.1-nano for local testing).
• (Optional) ngrok or similar tunnelling tool when exposing the server to ChatGPT.
• Familiarity with the latest OpenAI Model Context Protocol server specification (confirmed October 8, 2025) and the GPT-4.1 model family release notes to ensure endpoint compatibility.

Before starting an implementation sprint, re-verify the currently available GPT-4.1 model variants and pgvector/PostgreSQL release versions so documentation, configuration, and tests match production reality.

Running Locally

1. Ensure PostgreSQL (17.x with pgvector extension) is running on 127.0.0.1:5432 and the credentials in .env match.
2. Run migrations: npm run db:migrate (and per-project seeds with npm run seed:projects).
3. Start the MCP server in development mode: npm run dev (or npm start for the compiled build). The server listens on http://127.0.0.1:3333/mcp.
4. (Optional) Start the Cloudflare tunnel so external clients can reach the server: npm run stack:start -- --no-rebuild.
5. Exercise the tools locally with npm run tool:run -- db.list, search, etc., to verify results before connecting ChatGPT.

Scripts (current)

Script	Purpose
npm run dev	Execute the MCP server entrypoint via tsx server.ts for local development.
npm run build	Compile TypeScript to dist/ using the strict tsconfig.json.
npm start	Run the compiled server (node dist/server.js).
npm run lint	Lint the codebase with eslint (flat config + TypeScript rules).
npm run format	Check formatting with Prettier.
npm run format:fix	Apply Prettier formatting.
npm test	Run Jest unit/integration suites.
npm run ingest	Traverse sandbox/, invoke OpenAI embeddings, and upsert vectors into PostgreSQL.
npm run seed:projects	Create/refresh per-project databases from the contents of each sandbox/<project> directory (expects Postgres on 127.0.0.1).
npm run db:migrate	Apply Drizzle migrations to the database defined in .env.
npm run server:restart	Stop any server on port 3333, rebuild, and relaunch dist/server.js in the background.
npm run stack:start	Launch only the Cloudflare tunnel container that forwards to your locally running MCP server.
npm run stack:start:follow	Start the Cloudflare tunnel and follow its logs.
npm run stack:smoke	Perform an OPTIONS-based readiness probe against the MCP HTTP endpoint (http://127.0.0.1:3333/mcp).
npm run stack:logs	Tail Docker logs for the Cloudflare tunnel.

Tooling for ingestion is now available; see plan.md section 4 for remaining enhancements.

Version Checkpoint (October 8, 2025)

• OpenAI models: GPT-4.1, GPT-4.1 mini, and GPT-4.1 nano remain available for embeddings and summaries. GPT-4.1 mini now replaces GPT-4o mini in ChatGPT paid tiers while keeping the shared rate limits.
• PostgreSQL/pgvector: pgvector 0.7.4 (released August 5, 2024) is the latest generally available version and rolled into managed PostgreSQL 17 distributions during the October 6, 2025 maintenance window.
• MCP spec: Revision 2025-06-18 introduces OAuth2.1 requirements for HTTP transports and structured tool annotations; stdio handshake remains initialize → result → notifications/initialized.

Re-run these checks before shipping releases to confirm upstream expectations have not shifted.

File Structure (anticipated)

.
├── plan.md                         # Detailed task checklist (current planning doc)
├── README.md                       # Project overview and setup guidance
├── package.json                    # npm metadata and scripts (to be expanded)
├── tsconfig.json                   # Strict TypeScript compiler configuration (planned)
├── server.ts                       # MCP server entrypoint (placeholder)
├── src/
│   ├── types/
│   │   └── index.ts                # Canonical domain/tool schemas shared across modules
│   ├── ingestion/
│   │   ├── walker.ts               # File discovery helpers
│   │   ├── metadata.ts             # Tag parsing utilities
│   │   └── ingest.ts               # Embedding pipeline orchestrator
│   ├── db/
│   │   ├── pool.ts                 # PostgreSQL client factory with pgvector setup
│   │   └── schema.sql              # Table + index creation statements
│   ├── server/
│   │   ├── createMcpServer.ts      # Registers search/fetch tools and MCP resources
│   │   └── __tests__/createMcpServer.test.ts
│   └── prompts/
│       └── personas.ts             # Persona prompt/resource definitions
├── docs/
│   ├── testing.md                  # Manual + automated test checklist (planned)
│   └── architecture.md             # Extended design notes (planned)
└── sandbox/                        # Source files to ingest into the vector store (never runtime/server code)
    └── ...

Keep runtime/server logic out of sandbox/; only ingestible corpus files belong there to avoid the server indexing its own code.

Shared Types Library

All runtime and test modules import domain/tool schemas from the shared src/types package. This keeps JSON schemas, metadata interfaces, and persona definitions consistent across the server, ingestion pipeline, and future clients. Agents should add new contracts to src/types/index.ts (or a related module) and re-export them through a barrel so downstream code never redefines shapes locally. See AGENTS.md for the onboarding checklist covering this requirement.

Chunking Strategy

• Document summaries: one high-level chunk per file/persona to provide quick overviews for the orchestrator.
• Recursive base chunks: ~700-token segments with 15–20% overlap across all text formats to deliver balanced recall vs. cost.
• Format-aware slices: extra code-unit or heading-level chunks where structure matters, each tagged with granularity so queries can target the appropriate detail level.
• Persona definition files (stored as .json) are parsed before chunking so their key attributes (tone, constraints, goals) become searchable metadata while the raw JSON remains accessible for complete retrieval.
• Runtime chunk sizing and overlap thresholds are configurable via CHUNK_STRATEGY and related environment variables so experiments do not require code changes.

Coding Standards

• Write all runtime logic in TypeScript with strict compiler settings; JavaScript is reserved for generated artefacts only.
• Keep modules focused—follow the "one function or class per file" guideline and choose semantic filenames that describe intent.
• Centralise types within the shared library (src/types/), importing definitions instead of re-declaring them in features or tests.
• Practise TDD with Jest: add or update failing tests before implementation, and maintain passing suites with agreed coverage thresholds.
• Honour linting and formatting conventions once configuration lands; do not suppress rules without review.
• See docs/contributing.md for enforcement details, tooling commands, and review expectations.

Future Enhancements

• Evaluate OCR/ASR tooling so PDFs, images, and audio assets can be ingested alongside text sources.
• Harden Cloudflare tunnel configuration with IP allowlists or authentication for long-lived deployments.
• Explore richer UI components via Apps SDK output templates once core MCP tools stabilise.

Configuration

1. Copy .env.example to .env.
2. Update placeholder values:
   • OPENAI_API_KEY (required for embeddings + summaries).
   • PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD (defaults align with docker-compose.yml).

• OPENAI_EMBEDDING_MODEL / OPENAI_SUMMARY_MODEL (default to text-embedding-3-small and gpt-4.1-mini; bump to text-embedding-3-large if you prefer higher-fidelity vectors and can tolerate slower searches).
• CHUNK_STRATEGY (default recursive-700; adjust for experimentation).

3. Ensure the vector extension is installed (CREATE EXTENSION IF NOT EXISTS vector;). The bootstrap SQL in db/init/001-create-extension.sql handles this automatically for the Docker Compose flow.
4. OpenAI key must include access to the Embeddings API. If you see insufficient_quota errors, confirm the organization has an active billing plan and embeddings enabled.
5. Troubleshooting tips:
   • PostgreSQL connection failures: confirm docker compose ps shows mcp-vector-db healthy and that pg_isready succeeds.
   • Empty query results: run npm run ingest after adding files to sandbox/ so embeddings are available.
   • Large corpora: set CHUNK_STRATEGY to recursive-1200 (or similar) to reduce chunk counts while keeping overlap manageable.
   • Permission denied to create database when running npm run seed:projects:
     • Cause: the PGUSER role lacks the CREATEDB privilege but the seeding flow creates per-project databases.
     • Fix option A (recommended): grant CREATEDB to your app role, then run seeds:
       • psql -h 127.0.0.1 -U postgres -d postgres -c "ALTER ROLE mcp_user CREATEDB;"
     • Fix option B (no role change): set separate admin credentials used only for database creation and keep databases owned by your app role:
       • In .env, add PGADMIN_USER/PGADMIN_PASSWORD (a role with CREATEDB) and optionally PGADMIN_DATABASE=postgres.
       • Ensure PGOWNER is set to the desired owner (defaults to PGUSER).
       • Re-run npm run seed:projects.

Ingestion Workflow

1. Start PostgreSQL via docker compose up -d db and wait for the health check to pass.
2. Run npm run ingest to traverse sandbox/, parse bracketed metadata tags, chunk Markdown/TypeScript/YAML content, request embeddings from text-embedding-3-small, and upsert vectors into PostgreSQL.
3. The ingestion pipeline removes stale embeddings for files that were deleted or changed between runs, keeping the store aligned with sandbox/.

MCP Tools (current behaviour)

• db.list: Enumerates available project databases (derived from sandbox/<project> directories). Call this first to select the databaseId used by other tools.
• search: Accepts plain-text queries plus a databaseId, generates embeddings server-side, and returns sanitised sandbox results (ids, URIs, snippets, metadata) for the chosen project.
• fetch: Resolves chunk identifiers within the specified databaseId into full content payloads so the client can read the underlying files.
• sandbox.read: Reads an entire sandbox file by path for the given databaseId (limited to ≤256 KiB) and returns both text output and structured metadata.

Exposing the Local Server via Docker + Cloudflare

• docker-compose.yml now provisions only the Cloudflare tunnel (cloudflared). The PostgreSQL instance and MCP server run on your host (e.g., via Homebrew + npm run dev).
• Before the first run, copy your mcp-vector tunnel credential JSON into cloudflared/credentials/mcp-vector.json, export the origin certificate from cloudflared tunnel login to cloudflared/cert.pem, and ensure .env carries valid OpenAI/Postgres values.
• Use npm run stack:start for a detached launch or npm run stack:start:follow to stream Cloudflare logs during startup. The tunnel forwards traffic to http://host.docker.internal:3333, so keep the MCP server listening on that port locally.
• Run npm run stack:smoke after starting the server to send an OPTIONS probe to http://127.0.0.1:3333/mcp and confirm local readiness before exposing it through the tunnel.
• Check container state with docker compose ps, view logs with docker compose logs cloudflared, and shut everything down via docker compose down when you’re done.
• The MCP HTTP transport runs in stateless mode, so clients (including ChatGPT) do not need to send custom session headers—just issue the standard JSON-RPC initialize request to /mcp.

Development Roadmap (high level)

• Finalise requirements alignment (OpenAI MCP spec, acceptance criteria, Persona metadata expectations).
• Scaffold npm project, install dependencies, and commit base lint/test tooling.
• Implement PostgreSQL schema, helper library, and ingestion CLI.
• Build MCP server entrypoint and expose the canonical search/fetch tool pair.
• Add persona resources/prompts and metadata-driven workflow hints.
• Write tests (unit + integration) and validate with MCP Inspector.
• Ship documentation updates and prepare ChatGPT connector instructions.

Status

Planning in progress—per-project databases (db.list + database-scoped tools) and seeding scripts are live; follow plan.md for remaining chunking and relevance improvements.