forked from NOAA-EMC/global-workflow
-
Notifications
You must be signed in to change notification settings - Fork 0
MCP Health Status Comparative Report
Terry.McGuinness edited this page Jul 2, 2026
·
1 revision
Date: July 2, 2026
Servers compared: agentcore-mcp-rag (AWS-native, Python/FastMCP) vs eib-mcp-gateway (Node.js, Docker)
Method: Ran the six health/status utility tools on each server at the highest detail level the tools expose, then compared the results side by side.
This report is a live operational health/status snapshot. It complements the earlier design-level comparison in MCP-Parity-Assessment-SoC-Topology (which focused on response quality, data-gap analysis, and separation-of-concerns topology) by focusing strictly on what the two servers' own diagnostic tools report about themselves right now.
Each section presents a three-column table: AgentCore (agentcore-mcp-rag) | EIB Gateway (eib-mcp-gateway) | Comparative Analysis.
| Tool | AgentCore call | EIB Gateway call |
|---|---|---|
mcp_health_check |
deep=true, detailed=true, functional=true |
deep=true, detailed=true, functional=true |
get_server_info |
include_capabilities=true |
include_capabilities=true |
get_knowledge_base_status |
include_graph=true, include_vector=true |
include_graph=true, include_vector=true |
get_health_trend |
limit=10 |
limit=10 |
get_quality_metrics |
compare=true |
compare=true |
check_knowledge_integrity |
sample_size=50 |
sample_size=50 |
| Attribute | AgentCore (agentcore-mcp-rag) |
EIB Gateway (eib-mcp-gateway) |
Comparative Analysis |
|---|---|---|---|
| Server name / version | MDC MCP/RAG Server v1.0.0 | global-workflow-unified-mcp v3.6.2 | Different version lineages. Gateway is the mature, long-lived Node.js line (v3.x); AgentCore is the freshly-cut AWS-native rewrite (v1.0.0). The low AgentCore version reflects a new codebase, not lesser maturity. |
| Runtime | Python / FastMCP (AWS Bedrock AgentCore Runtime) | Node.js (Docker) | Fundamentally different stacks. AgentCore is AWS-managed serverless microVM; Gateway is a container. |
| Architecture label | 9 active modules | Week 2 Consolidated + v3.6.0 SOC Refactor (7 modules) | Gateway advertises an explicit consolidation/SoC refactor (8 duplicate tools eliminated, 21 unique tools). AgentCore reports a flat 9-module layout — the SoC-refactor topology recommendations from the parity assessment appear applied on the Gateway side. |
| Total tools | 52 | 51 | Near-identical surface. AgentCore's extra tool aligns with the 9th module split. Functionally the two expose the same capability families. |
| Multi-tenancy |
5 tenants (default gw) |
Single tenant (not tenant-aware) | Major differentiator. AgentCore is multi-tenant (gw, gw_sfs, gw_jedi_gfs, gw_v17, gw_gefs_v12), enabling branch-isolated knowledge bases. Gateway serves a single knowledge base. |
| Capabilities | Data Access connected · Vector available · Graph available | RAG enabled · GitHub enabled (no auth) · Workflow root auto-detected | Both fully wired to their data planes. Gateway explicitly reports GitHub is unauthenticated; AgentCore does not surface a GitHub-auth flag here but its GitHub functional probe passed. |
| Dimension | AgentCore (agentcore-mcp-rag) |
EIB Gateway (eib-mcp-gateway) |
Comparative Analysis |
|---|---|---|---|
| Overall status | HEALTHY (4/4 components) | HEALTHY (9/9 components) | Both healthy. The component counts differ only because each server groups its self-checks at a different granularity (AgentCore rolls up to 4 buckets; Gateway itemizes 9). Not a quality difference. |
| Component breakdown | Base (FastMCP), Utility (4 tools), Vector (21 indices), Graph (105,891 nodes / 4,729,093 rels) | Base (51 tools), Workflow Info (3), Code Analysis (4), Vector (15 indices), Graph (19,689 nodes), Semantic (15), Operational (3), GitHub (4), SDD (9) | AgentCore's health snapshot reports far more graph nodes/rels and more vector indices. Gateway's per-module itemization is more granular for spotting which subsystem is degraded. |
| Vector store | OpenSearch — 21 indices | ChromaDB — 15 indices | Different vector engines. AgentCore's OpenSearch carries more indices (dual embedding families + registry); Gateway's ChromaDB is leaner. |
| Graph store | Neptune (openCypher / SigV4 IAM) | Neo4j | Different graph engines. AgentCore = AWS-managed Neptune; Gateway = self-hosted Neo4j. |
| Data-plane probe | (rolled into component health) | ChromaDB heartbeat OK, 220,538 docs, sample query returned 10 results from global-workflow-docs-v8-0-0
|
Gateway surfaces an explicit end-to-end query probe in the health output. AgentCore validates the equivalent via its functional module tests (below). |
| Functional validation | 9/10 passed, 1 skipped, 0 failed | 6/6 passed, 0 failed | Both green. AgentCore runs more functional probes (10 modules incl. branch_isolation); its one skip is workflow_info because the workflow filesystem is not mounted. Gateway runs 6 tool-level probes, all pass. |
| Notable skip / caveat |
workflow_info SKIP — /mnt/workflow not mounted
|
none | AgentCore's workflow-filesystem-backed tools are degraded in this environment (EFS not mounted to the runtime). Gateway auto-detects its workflow root locally and has no equivalent gap. |
| Per-module latency (AgentCore) | semantic 198ms · code_analysis 43ms · graph_rag 62ms · ee2 192ms · operational 184ms · github 214ms · branch_isolation 407ms | (latency not itemized per functional test) | AgentCore exposes latency per functional probe, useful for spotting slow paths (branch_isolation is the slowest at 407ms — expected, it touches multiple tenants). Gateway reports pass/fail without per-test latency here (latency lives in get_quality_metrics). |
| Metric | AgentCore (OpenSearch) | EIB Gateway (ChromaDB) | Comparative Analysis |
|---|---|---|---|
| Collections | 16 | 15 | Comparable collection counts. |
| Total documents | 252,013 | 220,538 | AgentCore holds ~31.5K more documents (~14% larger corpus), largely due to dual embedding families and the SHA registry. |
| Embedding families | Titan 1024-dim + MPNet 768-dim (dual) | MPNet 768-dim (single) | Key differentiator. AgentCore maintains parallel Titan-1024 and MPNet-768 embeddings of the same content, enabling higher-dimensional semantic retrieval. Gateway is MPNet-768 only. |
| Largest code collection |
mdc-code-context-titan1024: 90,135 · mdc-code-context-mpnet768: 60,576 |
code-with-context-v8-0-0 / mdc-code-context-mpnet768: 60,574 |
The MPNet code corpora match almost exactly (60,576 vs 60,574) — strong evidence both ingested the same source tree. AgentCore adds a 90K-doc Titan code corpus on top. |
| Docs collections | workflow-docs titan1024 20,155 · mpnet768 22,498 · nova1024 150 | versioned: v8-0-0 22,498 · v8-1-0 20,511 · v8-2-0 23,624 | Gateway keeps multiple versioned doc snapshots (v8-0-0 → v8-2-0); AgentCore keeps one live set per embedding family. Gateway's versioning aids rollback; AgentCore's is leaner. |
| Community summaries | titan1024 2,113 · mpnet768 2,113 |
community-summaries 2,113 |
Identical count (2,113) — the hierarchical GraphRAG community layer is in parity across both servers. |
| Empty / placeholder collections | Several nova1024 collections at 0 docs (jjobs, community, ee2, code-context) | none reported empty | AgentCore has provisioned-but-unpopulated nova1024 collections (future embedding family, not yet ingested). Cosmetic; does not affect current retrieval. |
| Registry |
mdc-content-sha-registry: 52,754 |
(not exposed as a collection) | AgentCore surfaces a content-dedup SHA registry as a first-class collection; Gateway manages dedup internally. |
| Metric | AgentCore (Neptune) | EIB Gateway (Neo4j) | Comparative Analysis |
|---|---|---|---|
| Files | 17,273 | 17,273 | Exact match — both graphs index the identical file set. |
| Functions | 95,996 | 2,012 | Largest single divergence. AgentCore's function-level graph is ~48× denser. AgentCore materializes Fortran subroutines/functions and Python functions as first-class nodes; Gateway's Neo4j does not expand the Fortran call graph to the same depth. |
| Classes | 281 | 54 | AgentCore captures more class-like entities, consistent with its deeper code-graph ingestion. |
| Total nodes | 148,976 | 19,689 (from health probe; KB-status omits a total) | AgentCore's graph is ~7.5× larger by node count. This is the core "richness" gap the parity assessment flagged. |
| Total relationships | 4,555,408 | 4,220,211 | Closer than node counts suggest (~8% more on AgentCore). Both are multi-million-edge graphs dominated by CALLS. |
| CALLS | 3,407,104 | 3,306,540 | Within ~3% — the call graph is near-parity. |
| USES | 997,616 | 679,698 | AgentCore has ~47% more USES edges (deeper Fortran USE module resolution). |
| Relationship vocabulary | CALLS, USES, DEFINES, DEPENDS_ON_ENV, IMPORTS, EXPORTS, DEPENDS_ON, INVOKES, SOURCES, EXECUTES | CALLS, USES, MEMBER_OF, CONTAINS, DEPENDS_ON_ENV, SETS_ENV, DEFINES, IMPORTS, USES_ENV, EXPORTS | Overlapping but distinct edge taxonomies. Gateway has MEMBER_OF (91,815) + CONTAINS (35,109) + SETS_ENV/USES_ENV (community + env modeling). AgentCore has INVOKES/SOURCES/EXECUTES (cross-language shell→Fortran bridge). Each models environment and containment differently. |
| Node label breakdown | Function 87,610 · FortranSubroutine 27,941 · File 17,273 · FortranFunction 5,744 · FortranModule 4,800 · PythonFunction 2,642 · ShellScript 315 | (not label-broken-down; Shell graph reported separately) | AgentCore exposes a rich label taxonomy incl. 27,941 Fortran subroutines — absent from Gateway's summary. This is why AgentCore answers Fortran code-structure questions in more depth. |
| Shell graph detail | (folded into node labels: ShellScript 315) | Explicit: 628 scripts (178 J-Jobs, 12 ex, 12 ush), 2,760 env vars, SOURCES 1,237, INVOKES 2,115 | Gateway provides a dedicated Shell-graph breakdown (Phase 27B) with J-Job/ex/ush categorization; AgentCore reports 315 ShellScript nodes without the same operational breakdown. Gateway is stronger for shell-script operational queries. |
Note on internal count drift: AgentCore's
mcp_health_checksnapshot (105,891 nodes / 4,729,093 rels) differs from itsget_knowledge_base_status(148,976 nodes / 4,555,408 rels). The two tools count over different scopes/tenants — health rolls up a snapshot while KB-status queries the defaultgwtenant graph directly. Worth reconciling but not a data-integrity failure.
| Aspect | AgentCore | EIB Gateway | Comparative Analysis |
|---|---|---|---|
| Trend data | Empty — "No health history found" | Empty — "No health history found" |
Parity (both empty). Neither server has accumulated a persisted health-snapshot series yet. Each mcp_health_check run persists a snapshot to health_history.jsonl, so trends will populate on subsequent runs. This run seeds the first snapshot on both. |
| Metric | AgentCore | EIB Gateway | Comparative Analysis |
|---|---|---|---|
| Benchmark available |
No — "No benchmark results found" (quality_metrics.jsonl absent) |
Yes — 2026-03-11, corpus v1.0.0 (60 queries) | Biggest observability gap. AgentCore has not run its benchmark harness on the current runtime, so it reports no quality data. Gateway has a full benchmarked baseline. |
| Precision@5 | n/a | 0.71 | Gateway baseline: 0.71. AgentCore un-benchmarked — cannot compare retrieval quality quantitatively yet. |
| Recall@5 | n/a | 0.71 | " |
| MRR | n/a | 0.93 | Gateway's mean-reciprocal-rank is strong (0.93). |
| Coverage | n/a | 93% | " |
| Latency P50 / P95 | n/a | 42ms / 177ms | Gateway reports fast median retrieval. |
| Weakest category | n/a | Code Structure (P@5 0.40, MRR 0.70) | Gateway's known soft spot is code-structure retrieval — ironically the area AgentCore's denser graph should win, but AgentCore lacks a benchmark to prove it. |
| Strongest categories | n/a | EE2 (0.89), Semantic (0.88), Operational (0.83) | " |
| Regression vs prior | n/a | All categories IMPROVED except EE2 P50 (+6% DEGRADED) | Gateway trend is healthy; only a minor EE2 latency uptick. |
Action item: Run the AgentCore benchmark harness so the two servers can be compared on retrieval quality, not just infrastructure. Until then, quality parity is unproven for the AWS-native server.
| Check | AgentCore | EIB Gateway | Comparative Analysis |
|---|---|---|---|
| Overall | Issues detected | All checks passed | Gateway is clean; AgentCore surfaces two warnings. |
| Path Consistency | [WARN] 2/34 sampled docs have checkout-specific prefix | [OK] 0/0 sampled | AgentCore found a couple of docs with environment-specific path prefixes (a portability nit). Gateway sampled 0 (nothing flagged). |
| Orphaned Graph Nodes | [OK] 17,273 File nodes, 0/20 lack identity | [OK] 17,273 File nodes, 0/20 lack identity | Exact parity — both graphs are free of orphaned file nodes. |
| Stale Embeddings | [WARN] 12/12 sampled older than source (30-day threshold) | [OK] 0/0 current (git comparison unavailable → 30-day threshold) | AgentCore flags all 12 sampled embeddings as older than their source — a re-ingest/refresh signal. Gateway reports clean, but note it sampled 0/0 and fell back to a 30-day age heuristic (git comparison unavailable), so its "pass" is weaker evidence than it looks. |
| Coverage Gap | [SKIP] no Fortran files in /supported_repos/global-workflow
|
[SKIP] no Fortran files in supported_repos/global-workflow
|
Parity — both skip Fortran coverage because the on-disk checkout used for the probe has no Fortran (the Fortran lives in submodules not present at the probe path). |
| Theme | AgentCore (agentcore-mcp-rag) |
EIB Gateway (eib-mcp-gateway) |
Verdict |
|---|---|---|---|
| Platform | AWS-native (Neptune + OpenSearch + Bedrock Titan), serverless microVM | Node.js + Neo4j + ChromaDB, Docker container | Different generations of the same system; AgentCore is the strategic AWS target. |
| Health | HEALTHY 4/4, 9/10 functional | HEALTHY 9/9, 6/6 functional | Both healthy. Tie. |
| Multi-tenancy | 5 tenants, branch-isolated | Single tenant | AgentCore wins — branch isolation is a capability Gateway lacks. |
| Graph richness | 148,976 nodes, 95,996 functions, 27,941 Fortran subroutines | 19,689 nodes, 2,012 functions, no Fortran expansion | AgentCore wins on code-graph depth. |
| Shell-graph detail | 315 ShellScript nodes, no operational breakdown | 628 scripts w/ J-Job/ex/ush categorization | Gateway wins on shell operational modeling. |
| Vector corpus | 252K docs, dual Titan+MPNet | 220K docs, MPNet only + versioned snapshots | AgentCore larger + dual-embedding; Gateway keeps version history. Mixed. |
| Quality benchmark | Not generated | Full baseline (P@5 0.71, MRR 0.93) | Gateway wins — AgentCore quality is un-benchmarked. |
| Integrity | 2 warnings (path prefix, stale embeddings) | All pass (but weaker sampling) | Gateway cleaner on paper; AgentCore's warnings are actionable but its checks sampled more. |
| Workflow FS |
/mnt/workflow not mounted → 1 module degraded |
Auto-detected, no gap | Gateway wins in this environment. |
| Observability | Per-module latency, tenant table, functional matrix | Data-validation probe, quality benchmark, regression deltas | Complementary. AgentCore is stronger on live runtime introspection; Gateway on quality trend. |
Both servers are operationally healthy and expose the same ~51–52 tool surface over the same underlying Global Workflow source tree (file counts match exactly, and the MPNet code corpora are nearly identical). The meaningful differences are architectural and maturity-driven:
- AgentCore is the richer, multi-tenant, AWS-native platform with a far deeper code graph (esp. Fortran) and dual embeddings — but it has two open gaps in this environment: no quality benchmark generated and the workflow filesystem not mounted, plus minor integrity warnings (path prefixes, stale embeddings signalling a re-ingest is due).
- Gateway is the mature, single-tenant, benchmarked reference with clean integrity and better shell-operational graph detail — but a smaller graph, single embedding family, and no multi-tenancy.
Recommended follow-ups: (1) run the AgentCore benchmark harness to close the quality-comparison gap; (2) mount /mnt/workflow (EFS) to the AgentCore runtime to restore workflow_info; (3) trigger a re-ingest/embedding refresh on AgentCore to clear the stale-embedding warning; (4) reconcile the node/relationship count drift between AgentCore's mcp_health_check and get_knowledge_base_status.
Generated from live mcp_health_check, get_server_info, get_knowledge_base_status, get_health_trend, get_quality_metrics, and check_knowledge_integrity calls against both servers on July 2, 2026. See also MCP-Parity-Assessment-SoC-Topology for the design-level comparison and MDC-MCP-RAG-AWS-Architecture-v3 for the AgentCore deployment architecture.