forked from NOAA-EMC/global-workflow
-
Notifications
You must be signed in to change notification settings - Fork 0
FMS Analysis MCP Tool Effectiveness Report
Terry.McGuinness edited this page May 4, 2026
·
1 revision
Date: May 4, 2026
Query: "Where in the global-workflow infrastructure is FMS used?"
Platform: AgentCore Runtime v5 (Neptune 148K nodes, OpenSearch 206K docs)
Companion: FMS-Usage-in-Global-Workflow-Infrastructure
| # | Tool | Args (key) | Latency | Rating | Contribution |
|---|---|---|---|---|---|
| 1 | search_documentation |
"FMS Flexible Modeling System usage" | Fast | ⭐⭐⭐⭐ | Found the key UFS Weather Model docs on FMS diag_manager, diag_table, field_table |
| 2 | find_env_dependencies |
variable: "FMS" | Fast | ⭐⭐ | No shell-level FMS env var found — correct (FMS is Fortran-level, not shell) |
| 3 | get_code_context |
symbol: "FMS" | Fast | ⭐⭐⭐ | Symbol not found directly, but suggested _add_fms_cube_sphere_increments and cice_hist2fms — valuable leads |
| 4 | search_architecture |
"FMS diagnostic manager coupler" | Fast | ⭐⭐ | Returned community nodes but not directly FMS-related — architecture search less effective for library-level queries |
| 5 | find_similar_code |
"fms_mod diag_manager fms2_io" | Fast | ⭐ | No results above 0.7 threshold — similarity search not effective for module names |
| 6 | get_code_context |
symbol: "_add_fms_cube_sphere_increments" | 104ms | ⭐⭐⭐⭐ | Found the Python→Fortran DA bridge: finalize → to_fv3time → add_increments chain |
| 7 | search_documentation |
"FMS diag_table diag_manager output" | Fast | ⭐⭐⭐⭐⭐ | Excellent — returned detailed diag_table format, field_table format, diag_manager_nml, output file specs |
| 8 | find_similar_code |
"fms_init fms_end mpp_init" (threshold 0.5) | Fast | ⭐⭐⭐⭐ | Found 15 matches: MOM_io_infra.F90, MOM_diag_manager_infra.F90, mpas_dmpar.F, fort.F90 — confirmed MOM6 FMS integration |
| 9 | find_callers_callees |
function: "fms_init" | Fast | ⭐⭐⭐⭐⭐ | Key finding: 188 callers — quantified FMS pervasiveness across the codebase |
| 10 | find_callers_callees |
function: "diag_manager_init" | Fast | ⭐⭐ | 0 callers found — likely not ingested as a standalone symbol (called via FMS init chain) |
| 11 | find_callers_callees |
function: "fms_end" | Fast | ⭐⭐⭐⭐⭐ | Key finding: 316 callers — even more widely used than fms_init, confirmed MPI/domain/halo usage |
| 12 | get_code_context |
symbol: "cice_hist2fms" | 618ms | ⭐⭐⭐ | Found CICE→FMS format bridge in SOCA background utilities |
| 13 | describe_component |
"sorc/ufs_model.fd/FMS" | Fast | ⭐ | Component not found — describe_component searches workflow dirs, not deep submodule paths |
| 14 | get_code_context |
symbol: "fms_mod" | 89ms | ⭐⭐⭐⭐ | Confirmed FV3 climate nudging, remapping, variable state management as primary consumers |
| 15 | get_code_context |
symbol: "fms2_io_mod" | 147ms | ⭐⭐⭐⭐ | Found FV3 control init, namelist readers, JEDI FV3 interface — confirmed FMS2 I/O usage |
| 16 | get_code_context |
symbol: "diag_manager_mod" | 83ms | ⭐⭐⭐⭐ | Same consumer set as fms_mod — confirms tight coupling between FMS core and diagnostics |
| 17 | get_code_context |
symbol: "mpp_mod" | 86ms | ⭐⭐⭐⭐ | Confirmed message passing module used by same FV3 routines |
| 18 | get_code_context |
symbol: "constants_mod" | 53ms | ⭐⭐⭐⭐⭐ | Found UFS mediator usage: med_kind_mod, ufs_const_mod, all diagnostic phases — confirmed cross-component reach |
| 19 | explain_with_context |
"FMS role in UFS Weather Model" | Fast | ⭐ | Returned minimal content — RAG explanation thin for this topic |
Total tool calls: 19
Unique tools used: 7 (out of 51 available)
| Tool | Calls | Avg Rating | Notes |
|---|---|---|---|
find_callers_callees |
3 | 4.0 | Star performer — quantified fms_init (188) and fms_end (316) callers |
get_code_context |
7 | 3.6 | Excellent for tracing module dependencies and GGSR relationships |
The Neptune graph database was the primary value driver for this analysis. The caller/callee counts (188 and 316) are the most impactful data points in the entire report — they quantify FMS pervasiveness in a way that no text search could.
| Tool | Calls | Avg Rating | Notes |
|---|---|---|---|
search_documentation |
2 | 4.5 | Found detailed FMS diag_table docs, field_table format, output file specs |
find_similar_code |
2 | 2.5 | Mixed — failed at 0.7 threshold, succeeded at 0.5 for MOM6 FMS files |
search_architecture |
1 | 2.0 | Community-level search not ideal for library-level queries |
explain_with_context |
1 | 1.0 | Returned minimal content — needs tuning for infrastructure topics |
| Tool | Calls | Avg Rating | Notes |
|---|---|---|---|
find_env_dependencies |
1 | 2.0 | Correctly found no shell-level FMS env var (FMS is Fortran-level) |
describe_component |
1 | 1.0 | Can't reach deep submodule paths — limited to workflow-level dirs |
The following information was gathered via direct filesystem queries (find, grep) to supplement the MCP tool results:
| Method | Finding | MCP Gap |
|---|---|---|
find -iname "*fms*" -type d |
Found FMS1/, FMS2/, FMS_cap/ in MOM6 |
describe_component can't reach submodule paths |
find -iname "*fms*" -type f |
Found 21 FMS-related files across gdas.cd, gfs_utils.fd, MOM6 | No MCP tool for filename-pattern search in the repo |
grep -ri "FMS" .gitmodules |
FMS not a direct submodule (transitive via ufs_model.fd) | No submodule-aware MCP tool |
-
find_similar_code: Lower the default similarity threshold from 0.7 to 0.5 for Fortran module names — module names are short strings that don't embed well at high thresholds -
describe_component: Extend path search to include deep submodule paths (sorc/*/) not just top-level workflow directories -
explain_with_context: Needs richer context assembly for infrastructure/library topics — currently returns thin results -
New tool needed:
search_files_by_name— pattern-based filename search across the repo tree (thefindcommand equivalent) -
get_code_context: The[object Object]entries in GGSR output forfms_init/fms_endcallers suggest a serialization bug in the Neptune record-to-object conversion for nodes with complex properties
| Rating | Meaning |
|---|---|
| ⭐⭐⭐⭐⭐ | Excellent — provided key insights not available elsewhere |
| ⭐⭐⭐⭐ | Very good — directly contributed to the analysis |
| ⭐⭐⭐ | Good — provided useful supporting information |
| ⭐⭐ | Fair — returned results but limited direct value |
| ⭐ | Poor — returned no useful results or failed |
Report generated from 19 MCP tool calls across 7 unique tools on AgentCore Runtime v5.