Add JGLOBAL_ENKF_ECEN stale file handle failure analysis Root-cause analysis of a transient Lustre filesystem failure (ESTALE on prep_step) that caused a Hercules nightly CI run to fail in the EnKF ensemble recentering step. The compute work succeeded; the failure was in routine bookkeeping between two consecutive MPI invocations and the workflow had no retry logic for transient errors. Documents five recommended best practices: 1. prep_step retry wrapper for transient classes (ESTALE, EAGAIN, ENOMEM) 2. Preserve RUNDIR on non-zero exit (KEEPDATA_ON_FAILURE) 3. err_exit snapshots key files before destroying the working dir 4. CI-level auto-retry on well-known infrastructure signatures 5. Slurm step locality + sync between MPI runs Sourced via agentcore-mcp-rag MCP-RAG queries against the gw tenant (EE2 standards, operational guidance, code structure).

TerrenceMcGuinness-NOAA committed Jun 17, 2026