Skip to content

Per User RAG Isolation Developer Guide

Jump to bottom
Terry McGuinness edited this page Jul 20, 2026 · 3 revisions

Per-User RAG Isolation — Developer Guide (Wave 2: Personal Tenants)

Audience: SME developers (Anna, Anton, Brian, Georgios) with a personal scratch-clone of the MCP repo on the shared Parallel Works host.

TL;DR: You can iterate on EE2 standards, ingestion chunking, and search tuning inside your own personal tenant without disturbing the shared knowledge base. Your changes are namespaced — only you see them — until you merge to develop.

What Problem Does This Solve?

The shared MCP/RAG knowledge base (gw tenant) serves everyone:

  • 17 ChromaDB collections (220 K+ embedded documents)
  • 344,604 Neo4j nodes (graph of every Fortran/Python/Shell/CMake entity)
  • 53 MCP tools exposed via the Docker MCP Gateway on :18888

When you're on a feature branch and want to change how code gets embedded or how EE2 standards are chunked, you need a safe space to re-ingest and test — without breaking the baseline that everyone else queries.

Personal tenants give you that space on the same shared databases.

How It Works

┌───────────────────────────────────────────────────────────────────┐
│  Shared Infrastructure (always running, serves everyone)          │
│                                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐   │
│  │ ChromaDB     │  │ Neo4j        │  │ Docker MCP Gateway     │   │
│  │ :8080        │  │ :7474/:7687  │  │ :18888                 │   │
│  │              │  │              │  │ (shared 53-tool server)│   │
│  │ Collections: │  │ Labels:      │  │ Default tenant: gw     │   │
│  │  gw_*        │  │  Function    │  └────────────────────────┘   │
│  │  p_anna_*    │  │  P_ANNA_*    │                               │
│  │  p_anton_*   │  │  P_ANTON_*   │                               │
│  │  p_brian_*   │  │  P_BRIAN_*   │                               │
│  └──────────────┘  └──────────────┘                               │
└───────────────────────────────────────────────────────────────────┘
         ▲ write (your tenant only)   ▲ read-only (shared gw)
         │                            │
┌────────┼────────────────────────────┼──────────────────────────────┐
│  YOUR workspace                     │                              │
│  /mcp_rag_eib/SCRATCH_SPACE/<you>/eib-mcp-rag-server               │
│  Branch: feature/my-ee2-improvement │                              │
│                                     │                              │
│  ┌──────────────────────────────────┴───────────────────────┐      │
│  │ Your local MCP stdio server (runs from YOUR branch)      │      │
│  │                                                          │      │
│  │  • Queries gw tenant (read-only) — see baseline results  │      │
│  │  • Queries personal-<you> tenant — see YOUR changes      │      │
│  │  • --diff-tenant gw,personal-<you> — side-by-side compare│      │
│  │                                                          │      │
│  │ Connects to: bolt://localhost:7687, http://localhost:8080│      │
│  └──────────────────────────────────────────────────────────┘      │
│                                                                    │
│  VS Code Remote Tunnel → .vscode/mcp.json                          │
└────────────────────────────────────────────────────────────────────┘

Your personal tenant (personal-anna, personal-anton, etc.) writes to prefixed collections and labels on the same physical ChromaDB and Neo4j instances. The shared gw data is never touched.

Step-by-Step: EE2 Standards Iteration Example

1. Create your feature branch

cd /mcp_rag_eib/SCRATCH_SPACE/$USER/eib-mcp-rag-server
git checkout -b feature/ee2-chunk-reform

2. Create your personal tenant (one-time setup)

python -m mcp_server_python.scripts.create_personal_tenant $USER

This appends a new entry to tenants.yaml:

- tenant_id: personal-anna
  label_prefix: "P_ANNA_"
  index_prefix: "p_anna_"
  lifecycle: personal
  owner: Anna.Smoot

3. Make your changes

Edit the ingestion logic — for example, split EE2 standards by subsection instead of whole-document:

$EDITOR scripts/ingest_ee2_standards.py
# ... change chunking strategy ...

4. Re-ingest into YOUR tenant only

python scripts/ingest_ee2_standards.py --tenant personal-anna

This writes to p_anna_ee2-standards-v5-0-0-enhanced (ChromaDB) and any EE2-related graph nodes with P_ANNA_ prefix (Neo4j). The shared gw collections are untouched.

For the 34-document EE2 corpus, this takes seconds.

5. Test your changes

Query against your personal tenant:

# Using the MCP tool directly (via your local stdio server)
search_ee2_standards --query "module header comment format" --tenant personal-anna

6. Compare against baseline

Side-by-side comparison of your changes vs the shared baseline:

search_ee2_standards --diff-tenant gw,personal-anna \
    --query "module header comment format"

Output (example):

┌────────────────────────────────────────────────────────────────┐
│ Query: "module header comment format"                          │
├──────────────────────────────┬─────────────────────────────────┤
│ Baseline (gw)                │ Personal (personal-anna)        │
├──────────────────────────────┼─────────────────────────────────┤
│ 1. EE2-STD-003 (score 0.72)  │ 1. EE2-STD-003§2.1 (score 0.91) │
│ 2. EE2-STD-017 (score 0.65)  │ 2. EE2-STD-003§2.3 (score 0.88) │
│ 3. EE2-STD-022 (score 0.61)  │ 3. EE2-STD-017§1.2 (score 0.85) │
│ ...                          │ ...                             │
│ Relevant: 3/8                │ Relevant: 6/8 ← improvement     │
└──────────────────────────────┴─────────────────────────────────┘

7. Satisfied? Push and merge

git add -A
git commit -m "feat(ee2): split EE2 standards by subsection for better retrieval"
git push origin feature/ee2-chunk-reform
# → Open Merge Request on GitLab → review → merge to develop

After merge, the shared CI pipeline will re-ingest into the gw tenant with your improved chunking — benefiting all users.

8. Cleanup (automatic)

Your personal tenant data is automatically cleaned up after 30 days of inactivity. You can also manually remove it:

python -m mcp_server_python.scripts.delete_personal_tenant $USER

What You Can Safely Change on a Feature Branch

Change Type Example Personal Tenant Needed?
Tool query logic Edit ee2_compliance.py scoring No — just run your local MCP server against shared DBs (read-only)
Chunking / embedding rules Edit ingest_ee2_standards.py chunk size Yes — re-ingest into your personal tenant
New document sources Add a new standards PDF to the EE2 corpus Yes — ingest the new docs into your personal tenant
Search ranking / filtering Change max_results, add category filter No — tool code only, no data change
Compliance report templates Edit report formatting in generate_compliance_report No — tool code only

Key Rules

  1. Never ingest into the gw tenant from your feature branch. Always use --tenant personal-<you>.

  2. The shared gateway on :18888 always serves the gw baseline. Your VS Code MCP connection to the gateway gives you the production view. Your local stdio server (from your branch) gives you the personal view.

  3. Embeddings are deterministic. The mpnet768 model loads from a shared read-only cache. Your personal embeddings and the eventual shared re-ingest produce identical vectors for the same input text.

  4. Personal tenants are ephemeral. They exist to support your iteration loop. Once merged, the feature branch's logic runs on the shared tenant.

Your Workspace Layout

/mcp_rag_eib/SCRATCH_SPACE/<you>/
└── eib-mcp-rag-server/          ← YOUR clone (feature branches here)
    ├── .vscode/mcp.json          ← points at shared DBs + your local server
    ├── mcp_server_python/        ← tool code you're editing
    │   └── src/config/tenants.yaml  ← your personal tenant entry
    ├── scripts/                  ← ingestion scripts you're modifying
    └── supported_repos/          ← read-only submodules (don't modify)

The shared operator checkout at /mcp_rag_eib/eib-mcp-rag-server is read-only to you. You work exclusively in your scratch-space clone.

FAQ

Q: Can I accidentally break the shared knowledge base? A: No. Your local server runs with MCP_READ_ONLY=true for the gw tenant. Write operations are only permitted against your personal-* tenant. Even if you forget the flag, the adapter refuses writes to non-personal tenants from a user-owned process.

Q: How much disk does my personal tenant use? A: The EE2 corpus is 34 documents — trivial (< 1 MB). A full code-context personal ingest would use ~180 MB. There's a 500 MB soft limit per personal tenant.

Q: Do I need to restart anything after creating my personal tenant? A: Restart your local stdio server (not the shared gateway). The shared gateway doesn't need to know about personal tenants — they're for your local iteration only.

Q: What if my branch gets very old relative to develop? A: The system tracks the commit SHA at ingest time. If your personal tenant's data diverges > 100 commits from develop, check_knowledge_integrity will surface a WARN. Rebase your branch and re-ingest.

Q: Can I compare my personal tenant against another user's? A: Yes — --diff-tenant personal-anna,personal-anton works. Useful for collaborative iteration on the same feature.

Getting Started Checklist

  • Verify your scratch clone exists: ls ~/eib-mcp-rag-server (or /mcp_rag_eib/SCRATCH_SPACE/$USER/eib-mcp-rag-server)
  • Create a feature branch: git checkout -b feature/<your-change>
  • Create your personal tenant: python -m mcp_server_python.scripts.create_personal_tenant $USER
  • Make your changes to ingestion/tool code
  • Re-ingest: python scripts/ingest_ee2_standards.py --tenant personal-$USER
  • Compare: search_ee2_standards --diff-tenant gw,personal-$USER --query "…"
  • Push branch → open MR → merge → shared re-ingest happens automatically

Document created 2026-07-20 as part of Phase 74 (Per-User RAG Isolation). See the full technical spec: phase74_per_user_rag_isolation.md (linked from the gap analysis). For the shared-infrastructure health status, see Docker-MCP-Gateway-COTS-Gap-Analysis-2026-07-20.

Clone this wiki locally