Skip to content

Home

Jump to bottom
Terry McGuinness edited this page Nov 5, 2025 · 101 revisions

Global Workflow Wiki

Welcome to the NOAA Global Workflow technical wiki. This knowledge base documents solutions, configurations, and insights for operating and developing the Global Forecast System workflow.

🚀 Recent Updates: Embedding Model Upgrade (November 5, 2025)

Successfully upgraded the RAG system from all-MiniLM-L6-v2 (384 dimensions) to all-mpnet-base-v2 (768 dimensions), delivering a 50-100% improvement in semantic search quality for domain-specific queries. Empirical testing revealed the previous model achieved only 0.174-0.411 similarity scores on critical workflow terms (below the 0.5 acceptable threshold), prompting an immediate zero-cost upgrade. The new v4 collection achieved 73% completion (532/730 documents), enabling more accurate contextual AI assistance for global-workflow development and operations, with A/B testing and production cutover planned for completion. Full progress report.*

Development followed the Empirical Accuracy Principle: all technical claims verified through measurement rather than assumption, ensuring trustworthy AI-assisted development practices.

🎯 MCP Tool Architecture: 21-Tool Agentic AI Platform

NEW: Comprehensive documentation of the Model Context Protocol (MCP) server architecture that transforms GitHub Copilot from code completion to autonomous development assistance.

MCP_TOOL_ARCHITECTURE - Deep dive into the 21 specialized tools organized into 5 functional categories:

  • WorkflowInfoTools (3) - Foundation layer with instant structural awareness
  • CodeAnalysisTools (4) - Graph-based relationship intelligence via Neo4j
  • SemanticSearchTools (7) - RAG-enhanced knowledge retrieval with ChromaDB
  • OperationalTools (3) - Deep domain intelligence for HPC operations
  • GitHubTools (4) - Repository and project collaboration intelligence

Why This Matters: This architecture represents a paradigm shift from "AI that writes code" to "AI that understands systems." By combining filesystem analysis, graph databases (Neo4j), vector embeddings (ChromaDB), and semantic search, the MCP platform enables:

  • Autonomous research across documentation, code, and issues
  • Impact analysis before making changes (dependency graphs)
  • Compliance verification (EE2 standards) during development, not after
  • Operational intelligence with HPC-specific guidance
  • Collaborative awareness of ongoing work and project history

Configuration Modes:

  • full - All 21 tools (complete development environment)
  • core - 7 tools (minimal, no databases required)
  • rag - 17 tools (RAG without GitHub integration)

The Result: A fully-functional integrated agentic software development platform that doesn't just generate code - it understands architecture, follows standards, prevents breaking changes, and collaborates effectively. This is the future of weather model development at NOAA.

Documentation Status: Version 3.0.0 | Week 2 Consolidated Architecture | November 4, 2025

🔒 EE2 Compliance & Operational Readiness

NCEP Central Operations Compliance Analysis

Comprehensive EE2 compliance audits conducted using the MCP (Model Context Protocol) RAG infrastructure with hybrid semantic search (ChromaDB) and graph-based code analysis (Neo4j). These AI-assisted analyses examined hundreds of job scripts, execution scripts, and utility libraries to identify critical compliance gaps and provide production-ready remediation plans.

Global Workflow EE2 Analysis

  • EE2_COMPLIANCE_ANALYSIS_GLOBAL_WORKFLOW - 40+ page comprehensive audit of the global-workflow repository identifying top 5 critical compliance issues blocking operational deployment. Analysis covers 255+ files (172 job scripts, 83 execution scripts, utilities) with detailed remediation plans, production-ready code examples, and 14-week phased implementation timeline.

Key Findings:

  • Issue #1 (CRITICAL): Python error handling - 42 scripts lack try-except blocks
  • Issue #2 (HIGH): Shell error exits - && true pattern defeats error detection
  • Issue #3 (HIGH): Environment variable validation - ${PDY:-} defaults to empty
  • Issue #4 (MEDIUM-HIGH): Weak utility error handling - envsubst failures silent
  • Issue #5 (MEDIUM): Inconsistent set -e and missing trap handlers

Provenance: Generated via static analysis and MCP RAG tools examining NOAA-EMC/global-workflow fork using ChromaDB semantic search (730 docs) and Neo4j graph analysis (8709 relationships). Analysis date: November 3, 2025.

RRFS Workflow EE2 Analysis

  • EE2_COMPLIANCE_ANALYSIS_RRFS - Comprehensive EE2 compliance analysis of the RRFS (Rapid Refresh Forecast System) workflow repository. Examined 142+ files (26 jobs, 27 scripts, 35+ utilities, 54 Python modules) using MCP RAG tools. Key discovery: RRFS has better baseline compliance than global-workflow (consistent set -xue, custom error functions) but shares critical gaps. 10-week implementation plan with priority remediation targets.

Key Findings:

  • Issue #1 (CRITICAL): Missing err_chk function - All 26 job scripts call undefined function
  • Issue #2 (HIGH): Python error handling - Better structure than global-workflow but incomplete
  • Issue #3 (HIGH): Environment variable validation - Empty string defaults risk invalid paths
  • Issue #4 (MEDIUM-HIGH): No trap handlers - Resource leaks on failures
  • Issue #5 (MEDIUM): Insufficient error context - Good foundation needs enhancement

RRFS Advantages: Uses set -xue consistently (vs. set -e workarounds), custom print_err_msg_exit with caller context, filesystem operations use *_vrfy wrappers.

Provenance: Generated via MCP RAG hybrid analysis (semantic + graph) of NOAA-EMC/rrfs-workflow repository using ChromaDB vector search and Neo4j dependency mapping. Analysis date: November 3, 2025.

🚀 Advanced RAG & Graph Intelligence Infrastructure

Strategic Architecture Documents

The Global Workflow development infrastructure has evolved to incorporate state-of-the-art RAG (Retrieval-Augmented Generation) and Graph Database technologies, enabling sophisticated agentic AI capabilities for GFS software management and error analysis.

Core Infrastructure Documentation

  • README_PROVISIONING_V3.1_COMPLETE - Complete provisioning guide for the MCP RAG persistent infrastructure on ParallelWorks cloud platform. Covers ChromaDB 1.1.1 deployment, Node.js MCP server setup, LangFlow integration, and systemd service configuration for production-grade persistent storage architecture.

  • ENHANCED_INGESTION_ARCHITECTURE - Comprehensive design for Context7-inspired multi-source RAG ingestion across 50+ GFS submodules (3-5M LOC). Details the hybrid triple-store architecture combining ChromaDB (semantic search), Neo4j (graph relationships), and PostgreSQL (temporal data) for intelligent error diagnosis and code understanding.

  • CHROMADB_MIGRATION_COMPLETE - Technical documentation of ChromaDB 0.4.x to 1.1.1 migration, including API compatibility updates, Node.js client integration (chromadb@3.0.17), and resolution of embedding dimension mismatches for production stability.

Why Graph RAG for GFS Complexity?

The Challenge: The Global Forecast System represents one of the most complex software ecosystems in scientific computing:

  • 50+ interconnected repositories (UFS, GDAS, GSI, GOCART, MOM6, CICE, WW3, etc.)
  • 3-5 million lines of code across Fortran, Python, C/C++, and CMake
  • Deep dependency chains spanning atmospheric dynamics → ocean coupling → data assimilation → post-processing
  • Multi-component interactions that traditional documentation cannot capture

The Solution: Hybrid Graph + Vector RAG Architecture

Traditional vector-based RAG (ChromaDB alone) excels at semantic similarity but cannot answer structural questions:

  • ❌ "What components are affected if I change FV3 dynamics?"
  • ❌ "What's the dependency chain causing this compilation error?"
  • ❌ "Which CMakeLists.txt needs to link the GSW library?"
  • ❌ "Show me the call graph from model initialization to MPI communication"

Graph RAG (Neo4j + ChromaDB) enables these capabilities:

Error Analysis Workflow:
├─ Semantic Search (ChromaDB): Find similar errors and solutions
├─ Structural Analysis (Neo4j): Trace dependency chains and call graphs
├─ Temporal Context (PostgreSQL): Recent commits and regression patterns
└─ LLM Synthesis: Root cause + Fix instructions + Prevention recommendations

Agentic AI for GFS Software Management

The MCP (Model Context Protocol) server provides LLM agents with:

  1. Deep Code Understanding: Not just text search, but comprehension of component interactions
  2. Error Diagnosis: 10x faster debugging by combining similar past errors with structural impact analysis
  3. Impact Prediction: "What breaks if I change X?" before making changes
  4. Knowledge Retention: Institutional expertise captured in graph relationships
  5. Cross-Component Reasoning: Trace errors through UFS → GSI → GDAS → GFS pipeline

Result: Transform debugging from "search documentation and guess" to "query knowledge graph and know."

Implementation Status

  • ✅ ChromaDB 1.1.1: Production vector database operational
  • ✅ Node.js MCP Server: 17 tools for workflow management and RAG search
  • ✅ LangFlow UI: Visual workflow builder for RAG pipelines
  • 🚧 Neo4j Graph DB: Phase 0 POC approved, weekend implementation planned
  • 📋 Enhanced Ingestion: Multi-source ingestion pipeline designed for 50+ repos

Next Milestone: Neo4j proof-of-concept demonstrating dependency graph queries that ChromaDB cannot answer.

📖 NCEPLIBS-BUFR Error Catching Initiative

Core Documentation

  • PR673_Comprehensive_Analysis - Complete technical analysis of PR #673 which introduced error catching capability to NCEPLIBS-bufr. This 50+ page analysis covers the architectural design using setjmp/longjmp, implementation details across 51 files, code review insights, testing strategy, and operational impact for NOAA's weather forecasting infrastructure.

  • ERROR_CATCHING_IMPLEMENTATION_PLAN - Detailed 17-week implementation plan for extending error catching to 24 additional I/O routines following the PR #673 pattern. The plan divides work into 4 phases by complexity level, includes automated testing frameworks, CI/CD strategies, and comprehensive quality assurance checklists.

  • additional_io_routines_for_error_catching - Comprehensive inventory of 38 additional I/O routines organized into 7 complexity levels for systematic error catching implementation. This reference document provides technical details, implementation priorities, and success metrics for achieving complete API coverage in the BUFR library.

🧪 Background information of Cases used in the CTest Framework

The CTest framework provides self-contained test cases for validating individual workflow components. Each test creates an isolated environment with staged inputs from nightly stable baseline runs, enabling independent testing and validation.

C48 Fixed Atmosphere-Only Tests (ATM)

C48 Coupled System Tests (S2SW)

C48 Ensemble Tests (S2SW_gfs)

  • C48_S2SWA_gefs-gefs_fcst_mem001_seg0
  • GEFS ensemble member 001 coupled forecast (48-hour segment)
    • Implemented GEFS ensemble member 001 forecast test
    • 17 input files with unique two-cycle pattern:
      • 13 atmosphere ICs from current cycle (12Z)
      • 3 restart files from previous cycle (06Z)
      • 1 wave prep file from current cycle (12Z)
    • 24 output files (ensemble forecast outputs)
    • GEFS requires different source cycles for ICs vs restarts
    • Special handling for mem001/ subdirectory structure
  • C48_S2SWA_gefs-gefs_fcst_mem001_seg0.yaml

Framework Features:

  • Self-contained test environments with isolated EXPDIR
  • Input staging from STAGED_CTESTS (stable nightly runs)
  • Consistent naming convention: CASE-JOB.yaml
  • Comprehensive validation with input/output file verification

🔧 CI/CD & DevOps

GitLab CI/CD Pipeline

Jenkins Integration

GitHub & Jenkins Integration

🤖 AI/ML & Intelligent Tools

Model Context Protocol (MCP)

AI Development Tools

🌊 Workflow Management Systems

Rocoto Workflow Engine

CROW & EcFlow

🌐 Weather Modeling & Configuration

Model Configuration

💻 HPC System Administration

System Configuration

🔬 Research & Theory

Scientific Computing

🐛 Development & Debugging

Bug Fixes & Solutions

Development Process

📚 Quick Reference

Most Viewed Topics:

  • CI/CD Pipeline Architecture
  • Rocoto Workflow Management
  • MCP/RAG Integration
  • Jenkins Configuration
  • HPC System Setup

Latest Updates:

  • MCP Server RAG Enhancement
  • GitLab Multi-Host Architecture
  • AI-Assisted Development Tools

This wiki is actively maintained. Last organized: October 2025

Clone this wiki locally