Skip to content

Home

Jump to bottom
Tery McGuinness edited this page Dec 19, 2025 · 101 revisions

Global Workflow Wiki

Welcome to the NOAA Global Workflow technical wiki. This knowledge base documents solutions, configurations, and insights for operating and developing the Global Forecast System workflow.

📄 NEW: Docker MCP Gateway Technical Paper (December 15, 2025)

Docker MCP Gateway: Enabling MCP-as-a-Service for Enterprise AI Integration (PDF, 11 pages)

This comprehensive technical paper documents how Docker MCP Gateway transforms the Model Context Protocol from a single-client development tool into enterprise-ready multi-client infrastructure. The gateway bridges stdio and HTTP/SSE transports, enabling multiple AI clients (VS Code Copilot, Claude Desktop, LangFlow) to share common MCP tools simultaneously.

Key Topics Covered:

  • 🔄 MCP Transport Mechanisms - Why stdio limits single-client usage and how SSE enables network access
  • 🏗️ Gateway Architecture - Protocol bridging, session management, and container lifecycle
  • 🔒 Security Model - Network isolation, authentication, and resource limits
  • 🚀 NOAA Implementation - Production deployment with 32 tools, ChromaDB (14,854 docs), Neo4j (85,894 relationships)
  • 📋 Lessons Learned - Docker CE compatibility, label format requirements, network trade-offs

📥 Download PDF

Paper authored December 15, 2025 | NOAA EMC Global Workflow MCP Team

🧠 NEW: Phase 2 Semantic Annotation Architecture (December 4, 2025)

Breakthrough Achievement: 85% reduction in AI false positives through SME-driven semantic annotations embedded directly in technical standards documentation.

The Problem We Solved

AI-generated EE2 compliance recommendations suffered from systematic false positives—the AI was recommending patterns not actually required by NCEP operational standards (e.g., set -eu when only set -x is mandated). Traditional approaches required code changes for every correction.

The Solution: Semantic Annotations

Semantic annotations are machine-readable knowledge embedded in RST documentation that teach AI systems what patterns to recommend—and what to avoid:

.. mcp:anti_pattern:: adding_set_e_or_set_eu
   :severity: must_not
   :context: operational_scripts
   :sme_justification: Not present in EE2 standards or examples
   :evidence: standards.rst lines 588-595

Why This Matters for NOAA:

Before (Phase 1) After (Phase 2)
328 false positive violations 48 legitimate violations
Hard-coded rules in JavaScript SME-maintained RST annotations
Changes required programming Zero code changes to update rules
No evidence trail Complete traceability to EE2 source

Documentation Suite

  • PHASE_2_HYBRID_ARCHITECTURE_SPECIFICATION - Complete technical specification of the hybrid architecture that generates runtime configuration from semantic embeddings. Covers the 5-component pipeline (EE2 Standards → Annotations → ChromaDB → JSON Config → Scan Tool), validation results, and scalability analysis. Essential reading for understanding how semantic intelligence achieves runtime performance.

  • SME_Training_QuickStart - Practical 2-hour training guide for Subject Matter Experts on creating and reviewing semantic annotations. Includes linguistic framework (for translators/language experts), the 7 MCP directive types, and hands-on exercises. Enables domain experts to maintain compliance intelligence without programming.

  • SME Training QuickStart Guide (PDF) - Printable version of the training materials for offline use and in-person training sessions.

Architectural Innovation

The "hybrid" pattern combines the best of two worlds:

┌─────────────────────────────────────────────────────────────┐
│  BUILD TIME: Semantic Intelligence                          │
│  ChromaDB embeddings + Neo4j relationships → JSON Config    │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│  RUNTIME: Static Performance                                │
│  Load JSON once → O(1) lookup per file → Zero DB queries    │
└─────────────────────────────────────────────────────────────┘

Result: Semantic understanding WITHOUT runtime database queries. Scan 647 files in 12 seconds with full evidence traceability.

Impact on AI-Assisted Development

This architecture enables a new paradigm for expert-in-the-loop AI development:

  1. AI generates compliance recommendations using RAG-enhanced search
  2. SMEs review and identify false positives
  3. Annotations capture corrections in machine-readable form
  4. Pipeline regenerates configuration automatically
  5. AI learns without code changes

This is institutional knowledge preservation—capturing what experts know in a form that makes AI smarter.

� NEW: RAG Embedding Space Theory (December 19, 2025)

RAG_manifolds — Dimensional Conformality in Vector Databases: The Mathematical Foundation of RAG Embedding Spaces

A deep dive into why query and document embeddings must inhabit the same metric space for semantic search to work. On the surface, the cosine similarity formula is undergraduate linear algebra — but the 768-dimensional feature spaces encode recursive linguistic structures, emergent semantic geometry, and the holistic paradox of meaning encoded in vectors. Covers the mathematical foundations (SBERT, DPR, RAG papers), the superposition hypothesis, and the philosophical implications of meaning-as-geometry.

"The feature spaces are indeed a true enigma of recursive and holistic complexities — basic on the surface, infinitely deep upon reflection."

�🚀 Previous Update: Embedding Model Upgrade (November 5, 2025)

Successfully upgraded the RAG system from all-MiniLM-L6-v2 (384 dimensions) to all-mpnet-base-v2 (768 dimensions), delivering a 50-100% improvement in semantic search quality for domain-specific queries. Empirical testing revealed the previous model achieved only 0.174-0.411 similarity scores on critical workflow terms (below the 0.5 acceptable threshold), prompting an immediate zero-cost upgrade. The new v4 collection achieved 73% completion (532/730 documents), enabling more accurate contextual AI assistance for global-workflow development and operations, with A/B testing and production cutover planned for completion. Full progress report.

Development followed the Empirical Accuracy Principle: all technical claims verified through measurement rather than assumption, ensuring trustworthy AI-assisted development practices.

🎯 MCP Tool Architecture: 21-Tool Agentic AI Platform

NEW: Comprehensive documentation of the Model Context Protocol (MCP) server architecture that transforms GitHub Copilot from code completion to autonomous development assistance.

MCP_TOOL_ARCHITECTURE - Deep dive into the 21 specialized tools organized into 5 functional categories:

  • WorkflowInfoTools (3) - Foundation layer with instant structural awareness
  • CodeAnalysisTools (4) - Graph-based relationship intelligence via Neo4j
  • SemanticSearchTools (7) - RAG-enhanced knowledge retrieval with ChromaDB
  • OperationalTools (3) - Deep domain intelligence for HPC operations
  • GitHubTools (4) - Repository and project collaboration intelligence

Why This Matters: This architecture represents a paradigm shift from "AI that writes code" to "AI that understands systems." By combining filesystem analysis, graph databases (Neo4j), vector embeddings (ChromaDB), and semantic search, the MCP platform enables:

  • Autonomous research across documentation, code, and issues
  • Impact analysis before making changes (dependency graphs)
  • Compliance verification (EE2 standards) during development, not after
  • Operational intelligence with HPC-specific guidance
  • Collaborative awareness of ongoing work and project history

Configuration Modes:

  • full - All 21 tools (complete development environment)
  • core - 7 tools (minimal, no databases required)
  • rag - 17 tools (RAG without GitHub integration)

The Result: A fully-functional integrated agentic software development platform that doesn't just generate code - it understands architecture, follows standards, prevents breaking changes, and collaborates effectively. This is the future of weather model development at NOAA.

Documentation Status: Version 3.0.0 | Week 2 Consolidated Architecture | November 4, 2025

🔒 EE2 Compliance & Operational Readiness

NCEP Central Operations Compliance Analysis

Comprehensive EE2 compliance audits conducted using the MCP (Model Context Protocol) RAG infrastructure with hybrid semantic search (ChromaDB) and graph-based code analysis (Neo4j). These AI-assisted analyses examined hundreds of job scripts, execution scripts, and utility libraries to identify critical compliance gaps and provide production-ready remediation plans.

Global Workflow EE2 Analysis

  • EE2_COMPLIANCE_ANALYSIS_GLOBAL_WORKFLOW - 40+ page comprehensive audit of the global-workflow repository identifying top 5 critical compliance issues blocking operational deployment. Analysis covers 255+ files (172 job scripts, 83 execution scripts, utilities) with detailed remediation plans, production-ready code examples, and 14-week phased implementation timeline.

Key Findings:

  • Issue #1 (CRITICAL): Python error handling - 42 scripts lack try-except blocks
  • Issue #2 (HIGH): Shell error exits - && true pattern defeats error detection
  • Issue #3 (HIGH): Environment variable validation - ${PDY:-} defaults to empty
  • Issue #4 (MEDIUM-HIGH): Weak utility error handling - envsubst failures silent
  • Issue #5 (MEDIUM): Inconsistent set -e and missing trap handlers

Provenance: Generated via static analysis and MCP RAG tools examining NOAA-EMC/global-workflow fork using ChromaDB semantic search (730 docs) and Neo4j graph analysis (8709 relationships). Analysis date: November 3, 2025.

RRFS Workflow EE2 Analysis

  • EE2_COMPLIANCE_ANALYSIS_RRFS - Comprehensive EE2 compliance analysis of the RRFS (Rapid Refresh Forecast System) workflow repository. Examined 142+ files (26 jobs, 27 scripts, 35+ utilities, 54 Python modules) using MCP RAG tools. Key discovery: RRFS has better baseline compliance than global-workflow (consistent set -xue, custom error functions) but shares critical gaps. 10-week implementation plan with priority remediation targets.

Key Findings:

  • Issue #1 (CRITICAL): Missing err_chk function - All 26 job scripts call undefined function
  • Issue #2 (HIGH): Python error handling - Better structure than global-workflow but incomplete
  • Issue #3 (HIGH): Environment variable validation - Empty string defaults risk invalid paths
  • Issue #4 (MEDIUM-HIGH): No trap handlers - Resource leaks on failures
  • Issue #5 (MEDIUM): Insufficient error context - Good foundation needs enhancement

RRFS Advantages: Uses set -xue consistently (vs. set -e workarounds), custom print_err_msg_exit with caller context, filesystem operations use *_vrfy wrappers.

Provenance: Generated via MCP RAG hybrid analysis (semantic + graph) of NOAA-EMC/rrfs-workflow repository using ChromaDB vector search and Neo4j dependency mapping. Analysis date: November 3, 2025.

🚀 Advanced RAG & Graph Intelligence Infrastructure

Strategic Architecture Documents

The Global Workflow development infrastructure has evolved to incorporate state-of-the-art RAG (Retrieval-Augmented Generation) and Graph Database technologies, enabling sophisticated agentic AI capabilities for GFS software management and error analysis.

Core Infrastructure Documentation

  • README_PROVISIONING_V3.1_COMPLETE - Complete provisioning guide for the MCP RAG persistent infrastructure on ParallelWorks cloud platform. Covers ChromaDB 1.1.1 deployment, Node.js MCP server setup, LangFlow integration, and systemd service configuration for production-grade persistent storage architecture.

  • ENHANCED_INGESTION_ARCHITECTURE - Comprehensive design for Context7-inspired multi-source RAG ingestion across 50+ GFS submodules (3-5M LOC). Details the hybrid triple-store architecture combining ChromaDB (semantic search), Neo4j (graph relationships), and PostgreSQL (temporal data) for intelligent error diagnosis and code understanding.

  • CHROMADB_MIGRATION_COMPLETE - Technical documentation of ChromaDB 0.4.x to 1.1.1 migration, including API compatibility updates, Node.js client integration (chromadb@3.0.17), and resolution of embedding dimension mismatches for production stability.

Why Graph RAG for GFS Complexity?

The Challenge: The Global Forecast System represents one of the most complex software ecosystems in scientific computing:

  • 50+ interconnected repositories (UFS, GDAS, GSI, GOCART, MOM6, CICE, WW3, etc.)
  • 3-5 million lines of code across Fortran, Python, C/C++, and CMake
  • Deep dependency chains spanning atmospheric dynamics → ocean coupling → data assimilation → post-processing
  • Multi-component interactions that traditional documentation cannot capture

The Solution: Hybrid Graph + Vector RAG Architecture

Traditional vector-based RAG (ChromaDB alone) excels at semantic similarity but cannot answer structural questions:

  • ❌ "What components are affected if I change FV3 dynamics?"
  • ❌ "What's the dependency chain causing this compilation error?"
  • ❌ "Which CMakeLists.txt needs to link the GSW library?"
  • ❌ "Show me the call graph from model initialization to MPI communication"

Graph RAG (Neo4j + ChromaDB) enables these capabilities:

Error Analysis Workflow:
├─ Semantic Search (ChromaDB): Find similar errors and solutions
├─ Structural Analysis (Neo4j): Trace dependency chains and call graphs
├─ Temporal Context (PostgreSQL): Recent commits and regression patterns
└─ LLM Synthesis: Root cause + Fix instructions + Prevention recommendations

Agentic AI for GFS Software Management

The MCP (Model Context Protocol) server provides LLM agents with:

  1. Deep Code Understanding: Not just text search, but comprehension of component interactions
  2. Error Diagnosis: 10x faster debugging by combining similar past errors with structural impact analysis
  3. Impact Prediction: "What breaks if I change X?" before making changes
  4. Knowledge Retention: Institutional expertise captured in graph relationships
  5. Cross-Component Reasoning: Trace errors through UFS → GSI → GDAS → GFS pipeline

Result: Transform debugging from "search documentation and guess" to "query knowledge graph and know."

Implementation Status

  • ✅ ChromaDB 1.1.1: Production vector database operational
  • ✅ Node.js MCP Server: 17 tools for workflow management and RAG search
  • ✅ LangFlow UI: Visual workflow builder for RAG pipelines
  • 🚧 Neo4j Graph DB: Phase 0 POC approved, weekend implementation planned
  • 📋 Enhanced Ingestion: Multi-source ingestion pipeline designed for 50+ repos

Next Milestone: Neo4j proof-of-concept demonstrating dependency graph queries that ChromaDB cannot answer.

📖 NCEPLIBS-BUFR Error Catching Initiative

Core Documentation

  • PR673_Comprehensive_Analysis - Complete technical analysis of PR #673 which introduced error catching capability to NCEPLIBS-bufr. This 50+ page analysis covers the architectural design using setjmp/longjmp, implementation details across 51 files, code review insights, testing strategy, and operational impact for NOAA's weather forecasting infrastructure.

  • ERROR_CATCHING_IMPLEMENTATION_PLAN - Detailed 17-week implementation plan for extending error catching to 24 additional I/O routines following the PR #673 pattern. The plan divides work into 4 phases by complexity level, includes automated testing frameworks, CI/CD strategies, and comprehensive quality assurance checklists.

  • additional_io_routines_for_error_catching - Comprehensive inventory of 38 additional I/O routines organized into 7 complexity levels for systematic error catching implementation. This reference document provides technical details, implementation priorities, and success metrics for achieving complete API coverage in the BUFR library.

🧪 Background information of Cases used in the CTest Framework

The CTest framework provides self-contained test cases for validating individual workflow components. Each test creates an isolated environment with staged inputs from nightly stable baseline runs, enabling independent testing and validation.

C48 Fixed Atmosphere-Only Tests (ATM)

C48 Coupled System Tests (S2SW)

C48 Ensemble Tests (S2SW_gfs)

  • C48_S2SWA_gefs-gefs_fcst_mem001_seg0
  • GEFS ensemble member 001 coupled forecast (48-hour segment)
    • Implemented GEFS ensemble member 001 forecast test
    • 17 input files with unique two-cycle pattern:
      • 13 atmosphere ICs from current cycle (12Z)
      • 3 restart files from previous cycle (06Z)
      • 1 wave prep file from current cycle (12Z)
    • 24 output files (ensemble forecast outputs)
    • GEFS requires different source cycles for ICs vs restarts
    • Special handling for mem001/ subdirectory structure
  • C48_S2SWA_gefs-gefs_fcst_mem001_seg0.yaml

Framework Features:

  • Self-contained test environments with isolated EXPDIR
  • Input staging from STAGED_CTESTS (stable nightly runs)
  • Consistent naming convention: CASE-JOB.yaml
  • Comprehensive validation with input/output file verification

🔧 CI/CD & DevOps

GitLab CI/CD Pipeline

Jenkins Integration

GitHub & Jenkins Integration

🤖 AI/ML & Intelligent Tools

Model Context Protocol (MCP)

AI Development Tools

🌊 Workflow Management Systems

Rocoto Workflow Engine

CROW & EcFlow

🌐 Weather Modeling & Configuration

Model Configuration

💻 HPC System Administration

System Configuration

🔬 Research & Theory

Scientific Computing

🐛 Development & Debugging

Bug Fixes & Solutions

Development Process

📚 Quick Reference

Most Viewed Topics:

  • CI/CD Pipeline Architecture
  • Rocoto Workflow Management
  • MCP/RAG Integration
  • Jenkins Configuration
  • HPC System Setup

Latest Updates:

  • Phase 2 Semantic Annotation Architecture (December 2025)
  • SME Training for Semantic Annotations (December 2025)
  • Hybrid Build-Time/Runtime Compliance Validation
  • MCP Server RAG Enhancement
  • AI-Assisted Development Tools

This wiki is actively maintained. Last organized: December 2025

Clone this wiki locally