forked from NOAA-EMC/global-workflow
-
Notifications
You must be signed in to change notification settings - Fork 0
MCP RAG Platform June July 2026 Milestone Report
Terry McGuinness edited this page Jul 6, 2026
·
1 revision
Period: June 1 – July 6, 2026 (115 commits, branch develop_aws_startpoint)
Server: MDC MCP/RAG v1.0.0 · 53 tools · 9 active modules · 5 tenants
Status at close of period: Gateway healthy — 11/11 functional tests pass, 4/4 components healthy
Over five weeks the MDC MCP-RAG platform completed its transition from a single-tenant, AWS-only research prototype to a production-grade, five-tenant, dual-backend (COTS/AWS) system running natively on Parallel Works and exposing a 53-tool MCP endpoint via the Docker MCP Gateway (port 18888, Streamable HTTP). Ten numbered phases and six multi-phase gap-closure sprints were completed. The period ended with the retirement of the legacy static Node.js container and full functional parity between the Python stdio server and the gateway-spawned Python container.
Phases: Gap A (tenant-id tool exposure), provisioning v4.1.0 multi-user
| Item | Detail |
|---|---|
| Tenant-id wired to 24 tools | All semantic search, code analysis, graph-RAG, EE2, and operational tools now accept tenant_id; isolates query results per branch |
| Cross-tenant leakage fix | Added :Label qualifiers to all Cypher MATCH patterns preventing gw data bleeding into gw_v17 or gw_sfs results |
| Five tenants registered |
gw (develop), gw_sfs (dev/sfs), gw_jedi_gfs (dev/jedi-gfs), gw_v17 (dev/gfs.v17), gw_gefs_v12 (release/gefs_v12) |
| Multi-user provisioning |
00-users.sh extended to 5 workflow-developer accounts; .bashrc template, uv/uvx, git-branch prompt, scratch workspace per-user |
| GitHub token from Secrets Manager | AWS-standard startup path; eliminates plaintext PAT in env vars |
| New RAG doc sources | ECMWF Atlas + JEDI Academy 2021-06/10 added → 58 sources, 56 enabled |
Phases: Gap A close, Gap B (graph port), Gaps D, E, F
| Gap | Fix | Release |
|---|---|---|
| Gap A | Tenant-id tool exposure deployed (v22); all 24 tools scoped | — |
| Gap D | Bracket-aware tenant label rewriter for complex Cypher queries | 8.31.0 |
| Gap E | Tenant-scope added to label-less graph queries (prevents global fan-out) | 8.32.0 |
| Gap F | Fortran parse fallback (regex when AST fails) + parallel parse runner + streaming OOM fix | 8.33.0 / 8.34.0 |
| Gap B | Graph port series — Shell ops, workflow structure, and Fortran AST parsers ported to Python pipeline; 9 tasks completed | — |
| Gap G | Bounded graph traversal deployed (v32); gap closed | — |
| Gap I | v17 knn-vector reindex (float → knn_vector field type); gap resolved | — |
| OpenSearch Bug 1 |
resolve_index called before tenant-prefix applied in adapter query |
8.36.2 |
| OpenSearch Bug 2 |
_render_vector_status_block not tenant-scoped in knowledge-base status |
8.36.2 |
| Health-check Bug 1 |
_parse_iso_ts UTC fallback + skip-count reconciliation (R3.5) |
8.36.1 |
| Health-check Bug 2 |
_smoke_workflow_info SKIPs (non-fatal) when workflow mount missing |
8.36.1 |
| Graceful missing-index | 4 tools now return [INFO] Skip_Block instead of error when index absent |
8.36.3 |
Experiment directories imported: 17 clean EXPDIR entries (develop) + 9 from dev/gfs.v17 for workflow-structure ingestion.
Phases: NIH Sandbox Cost Control, Cross-Platform Data Persistence
| Item | Detail |
|---|---|
| CDK sleep/wake IaC — Waves 0–6 | Full CDK app covering EC2, Neptune, OpenSearch, AgentCore, and NAT tier hibernation/wake; Wave 3 state machine + CLI orchestrator composes all tiers |
| Drift detection | Wave 2 Group C: detects resource drift from expected sleep state; wake probe verifies services healthy after resume |
| Quickstart hibernate/wake scripts | Operator-facing CLI scripts for end-of-day sleep and morning wake of the entire NIH sandbox |
| Symmetric export/restore adapters | Bidirectional AWS ↔ COTS pipeline: Neptune → Parquet → Neo4j and ChromaDB → Parquet → OpenSearch (and reverse); 4-mode operation |
| Portable export live artifacts |
neptune_reader fixed for label-family streaming; sync_to_pw.sh + s3_cross_account_copy.py for PW bucket sync; 20 ingestion run reports (May 28 – Jun 11) |
| IAM policy tightened | CDK orchestrator IAM policy reviewed and least-privileged per pre-deploy review |
Phases: Python MCP PW legacy-backend integration, Phase 60 (code awareness parity + branch validation)
| Item | Detail |
|---|---|
| COTS backend integration |
mcp_server_python running natively on PW Rocky 9 against local ChromaDB + Neo4j via DB_BACKEND=cots; all probes backend-agnostic |
| Backend-agnostic graph health probe | Neo4j per-label node count replaces AWS-specific OpenSearch probe — works on both backends |
| Code awareness parity |
code_awareness and branch_isolation functional test modules added; both pass (411 ms, 248 ms) |
| Phase 61: Configurable workflow mount |
MCP_WORKFLOW_MOUNT env var + .pw_workflow_mount/ symlink farm; all 5 tenant workflow_root paths resolve on PW; get_workflow_structure/list_job_scripts/find_env_dependencies fully operational |
| Docker Gateway auth fix | Authorization token header wired correctly in streaming transport |
| SDD steering additions | Commit message standards + tool-count reconciliation directives added to .kiro/steering/
|
Phase: Phase 62
| Item | Detail |
|---|---|
extract_ci_error_signal tool |
New MCP tool distils multi-MB CI log files to an 8 KB LLM-consumable footprint; applies noise filtering (bash env dumps, module chatter) and captures failure signals with tail tracebacks + exit codes |
ErrorRecord schema |
Standardized payload schema for structured failure extraction |
| Ordered failure taxonomy | Classifier applies priority-ordered failure taxonomy before signal capture |
| Reporting directive |
.kiro/steering/13-ci-error-reporting-policy.md mandates MCP tooling for all CI log analysis |
| Module registration fix |
error_analysis module registered in KNOWN_MODULES (post-Phase 62 hotfix) |
Phase: Phase 63a
| Item | Detail |
|---|---|
DB_BACKEND=cots canonical |
Renames the on-prem backend value from legacy to cots (Commercial Off-The-Shelf: ChromaDB + Neo4j); accurately names the stack at the same conceptual level as aws
|
| Deprecation shim |
DB_BACKEND=legacy still accepted through one release cycle with a single [WARN] per process; auto-maps to cots
|
| Removal scheduled | Phase 64 — is_legacy() alias and legacy shim removed |
| Scope | Python server, Node scripts, .vscode/mcp.json example, unit tests — 15+ files updated |
Phase: Phase 63b — 11/11 functional tests passing
| Item | Detail |
|---|---|
| New gateway image |
eib-mcp-rag-python:latest (x86_64 Python build) replaces eib-mcp-rag:latest (Node.js) in the catalog |
| Tool parity | 53 tools / 9 modules / 5 tenants — matches stdio server exactly |
| COTS dependencies baked |
chromadb==1.3.4, neo4j==5.25.0, sentence-transformers==5.1.2, torch==2.9.0+cpu (CPU index); Spack-pinned |
| Offline mpnet768 embeddings |
HF_HUB_OFFLINE=1; 768-dim mpnet encoder loads from host HF cache mounted read-only at /app/.hf_cache
|
| Relative symlinks |
.pw_workflow_mount uses realpath --relative-to so links resolve identically on host and inside container |
GitHub token via --secrets
|
GITHUB_TOKEN wired through docker-mcp secrets: catalog block + mcp-gateway-launch.sh wrapper sourcing ~/.config/eib-mcp/secrets.env onto tmpfs |
| Health verification | 4/4 components healthy; 15 collections / 220,538 docs; 108,280 nodes / 4,220,211 relationships; all 5 tenants workflow_root reachable = yes
|
ExecStopPost shell-pipe note |
Harmless journal error (unknown shorthand flag: -r) on service stop; wrap in /bin/sh -c '...' deferred to Phase 63x cleanup |
Phase: Phase 63c
| Item | Detail |
|---|---|
mcp-rag.service retired |
Stopped, disabled; no longer a prerequisite of mcp-gateway.service
|
| 8 GB / 4 CPU freed |
eib-mcp-rag-static container (Node.js, single-tenant, 51 tools) no longer running idle |
| Dependency rewrite |
mcp-gateway.service [Unit] Requires= rewritten from mcp-rag.service → docker.service (drop-ins cannot reset dependency lists; base unit edited directly; backup at .bak-phase63c) |
| Cron resurrection gap |
/etc/cron.d/mcp-health (*/5 * * * *) was silently restarting mcp-rag.service after disable; Rocky 9 crond reads dotted filenames so rename to .disabled was insufficient — file deleted |
| In-tree deprecation |
12-static-mode-gateway.sh short-circuits exit 0; SETUP/cron.d/mcp-health, SETUP/bin/health-check.sh, SETUP/bin/deploy-static-gateway.sh all marked DEPRECATED with rollback guards |
| Rollback preserved | Node image eib-mcp-rag:latest (id cc69b32e3c67) retained locally; one-line catalog revert restores the prior topology |
Phase: Phase 65 (spec; implementation deferred to week of Jul 6)
| Item | Detail |
|---|---|
| Phase 65 IaC spec |
DevHostStack (CDK) to replace console-created c6g.xlarge with IaC-managed r7g.2xlarge ARM AL2023 AMI; EFS-mounted, wired to existing Neptune/OpenSearch/Bedrock stacks |
| Kiro CLI musl fix | AL2023 glibc 2.34 incompatible with glibc-2.38/2.39 kiro-cli update build; SETUP/update-kiro-cli-musl.sh pulls the statically-linked musl build instead |
| Dev tunnel utility | Helper script for managing VS Code dev tunnel connections added |
| Metric | Value |
|---|---|
| Tools | 53 (9 active modules) |
| Tenants | 5 (default: gw) |
| ChromaDB collections | 15 |
| Total vector documents | 220,538 |
| Neo4j nodes | 225,836 |
| Neo4j relationships | 4,049,454 |
| RAG doc sources | 58 (56 enabled) |
| Functional test result | 11/11 pass |
| Gateway uptime |
mcp-gateway.service active + enabled; Restart=always
|
| SDD sessions | 65 workflows, 48 sessions (34 completed, 5 abandoned) |
| Gap | Description | Priority |
|---|---|---|
| Non-default tenant ingestion |
gw_sfs, gw_jedi_gfs, gw_gefs_v12 have zero vectors + zero graph; gw_v17 has graph but no vectors |
High |
check_knowledge_integrity scoping |
Coverage-gap and orphaned-node checks are not tenant-scoped; false-positive "all passed" on empty tenants | Medium |
| Quality benchmark |
quality_metrics.jsonl absent; get_quality_metrics returns no data |
Medium |
Phase 63c ExecStopPost
|
Shell pipe | xargs -r produces harmless journal error on gateway stop |
Low |
| Phase 64 cleanup | Remove DB_BACKEND=legacy shim and is_legacy() alias |
Low |
Generated 2026-07-06 from git log --since=2026-06-01 develop_aws_startpoint (115 commits) and live Docker MCP Gateway health tools.