Skip to content

Home

Jump to bottom
Terry McGuinness edited this page Nov 3, 2025 · 101 revisions

Global Workflow Wiki

Welcome to the NOAA Global Workflow technical wiki. This knowledge base documents solutions, configurations, and insights for operating and developing the Global Forecast System workflow.

🚀 Advanced RAG & Graph Intelligence Infrastructure

Strategic Architecture Documents

The Global Workflow development infrastructure has evolved to incorporate state-of-the-art RAG (Retrieval-Augmented Generation) and Graph Database technologies, enabling sophisticated agentic AI capabilities for GFS software management and error analysis.

Core Infrastructure Documentation

  • README_PROVISIONING_V3.1_COMPLETE - Complete provisioning guide for the MCP RAG persistent infrastructure on ParallelWorks cloud platform. Covers ChromaDB 1.1.1 deployment, Node.js MCP server setup, LangFlow integration, and systemd service configuration for production-grade persistent storage architecture.

  • ENHANCED_INGESTION_ARCHITECTURE - Comprehensive design for Context7-inspired multi-source RAG ingestion across 50+ GFS submodules (3-5M LOC). Details the hybrid triple-store architecture combining ChromaDB (semantic search), Neo4j (graph relationships), and PostgreSQL (temporal data) for intelligent error diagnosis and code understanding.

  • CHROMADB_MIGRATION_COMPLETE - Technical documentation of ChromaDB 0.4.x to 1.1.1 migration, including API compatibility updates, Node.js client integration (chromadb@3.0.17), and resolution of embedding dimension mismatches for production stability.

Why Graph RAG for GFS Complexity?

The Challenge: The Global Forecast System represents one of the most complex software ecosystems in scientific computing:

  • 50+ interconnected repositories (UFS, GDAS, GSI, GOCART, MOM6, CICE, WW3, etc.)
  • 3-5 million lines of code across Fortran, Python, C/C++, and CMake
  • Deep dependency chains spanning atmospheric dynamics → ocean coupling → data assimilation → post-processing
  • Multi-component interactions that traditional documentation cannot capture

The Solution: Hybrid Graph + Vector RAG Architecture

Traditional vector-based RAG (ChromaDB alone) excels at semantic similarity but cannot answer structural questions:

  • ❌ "What components are affected if I change FV3 dynamics?"
  • ❌ "What's the dependency chain causing this compilation error?"
  • ❌ "Which CMakeLists.txt needs to link the GSW library?"
  • ❌ "Show me the call graph from model initialization to MPI communication"

Graph RAG (Neo4j + ChromaDB) enables these capabilities:

Error Analysis Workflow:
├─ Semantic Search (ChromaDB): Find similar errors and solutions
├─ Structural Analysis (Neo4j): Trace dependency chains and call graphs
├─ Temporal Context (PostgreSQL): Recent commits and regression patterns
└─ LLM Synthesis: Root cause + Fix instructions + Prevention recommendations

Agentic AI for GFS Software Management

The MCP (Model Context Protocol) server provides LLM agents with:

  1. Deep Code Understanding: Not just text search, but comprehension of component interactions
  2. Error Diagnosis: 10x faster debugging by combining similar past errors with structural impact analysis
  3. Impact Prediction: "What breaks if I change X?" before making changes
  4. Knowledge Retention: Institutional expertise captured in graph relationships
  5. Cross-Component Reasoning: Trace errors through UFS → GSI → GDAS → GFS pipeline

Result: Transform debugging from "search documentation and guess" to "query knowledge graph and know."

Implementation Status

  • ✅ ChromaDB 1.1.1: Production vector database operational
  • ✅ Node.js MCP Server: 17 tools for workflow management and RAG search
  • ✅ LangFlow UI: Visual workflow builder for RAG pipelines
  • 🚧 Neo4j Graph DB: Phase 0 POC approved, weekend implementation planned
  • 📋 Enhanced Ingestion: Multi-source ingestion pipeline designed for 50+ repos

Next Milestone: Neo4j proof-of-concept demonstrating dependency graph queries that ChromaDB cannot answer.

🔒 EE2 Compliance & Operational Readiness

NCEP Central Operations Compliance

  • EE2_COMPLIANCE_ANALYSIS - Comprehensive EE2 compliance audit of global-workflow repository (November 2025). This 40+ page analysis identifies the top 5 critical compliance issues blocking operational deployment, with detailed remediation plans and production-ready code examples. Generated via AI-assisted analysis using MCP RAG tools examining 255+ files (172 job scripts, 83 execution scripts, utility libraries). Covers Python error handling (CRITICAL), shell script error exits (HIGH), environment variable validation (HIGH), utility function robustness (MEDIUM-HIGH), and trap handler implementation (MEDIUM). Includes 14-week phased implementation plan, testing strategy, and success metrics. Provenance: Created from static analysis of NOAA-EMC/global-workflow fork using hybrid semantic search (ChromaDB) and graph-based code analysis (Neo4j) via the MCP server infrastructure.

📖 NCEPLIBS-BUFR Error Catching Initiative

Core Documentation

  • PR673_Comprehensive_Analysis - Complete technical analysis of PR #673 which introduced error catching capability to NCEPLIBS-bufr. This 50+ page analysis covers the architectural design using setjmp/longjmp, implementation details across 51 files, code review insights, testing strategy, and operational impact for NOAA's weather forecasting infrastructure.

  • ERROR_CATCHING_IMPLEMENTATION_PLAN - Detailed 17-week implementation plan for extending error catching to 24 additional I/O routines following the PR #673 pattern. The plan divides work into 4 phases by complexity level, includes automated testing frameworks, CI/CD strategies, and comprehensive quality assurance checklists.

  • additional_io_routines_for_error_catching - Comprehensive inventory of 38 additional I/O routines organized into 7 complexity levels for systematic error catching implementation. This reference document provides technical details, implementation priorities, and success metrics for achieving complete API coverage in the BUFR library.

🧪 Background information of Cases used in the CTest Framework

The CTest framework provides self-contained test cases for validating individual workflow components. Each test creates an isolated environment with staged inputs from nightly stable baseline runs, enabling independent testing and validation.

C48 Fixed Atmosphere-Only Tests (ATM)

C48 Coupled System Tests (S2SW)

C48 Ensemble Tests (S2SW_gfs)

  • C48_S2SWA_gefs-gefs_fcst_mem001_seg0
  • GEFS ensemble member 001 coupled forecast (48-hour segment)
    • Implemented GEFS ensemble member 001 forecast test
    • 17 input files with unique two-cycle pattern:
      • 13 atmosphere ICs from current cycle (12Z)
      • 3 restart files from previous cycle (06Z)
      • 1 wave prep file from current cycle (12Z)
    • 24 output files (ensemble forecast outputs)
    • GEFS requires different source cycles for ICs vs restarts
    • Special handling for mem001/ subdirectory structure
  • C48_S2SWA_gefs-gefs_fcst_mem001_seg0.yaml

Framework Features:

  • Self-contained test environments with isolated EXPDIR
  • Input staging from STAGED_CTESTS (stable nightly runs)
  • Consistent naming convention: CASE-JOB.yaml
  • Comprehensive validation with input/output file verification

🔧 CI/CD & DevOps

GitLab CI/CD Pipeline

Jenkins Integration

GitHub & Jenkins Integration

🤖 AI/ML & Intelligent Tools

Model Context Protocol (MCP)

AI Development Tools

🌊 Workflow Management Systems

Rocoto Workflow Engine

CROW & EcFlow

🌐 Weather Modeling & Configuration

Model Configuration

💻 HPC System Administration

System Configuration

🔬 Research & Theory

Scientific Computing

🐛 Development & Debugging

Bug Fixes & Solutions

Development Process

📚 Quick Reference

Most Viewed Topics:

  • CI/CD Pipeline Architecture
  • Rocoto Workflow Management
  • MCP/RAG Integration
  • Jenkins Configuration
  • HPC System Setup

Latest Updates:

  • MCP Server RAG Enhancement
  • GitLab Multi-Host Architecture
  • AI-Assisted Development Tools

This wiki is actively maintained. Last organized: October 2025

Clone this wiki locally