Should WE2E tests with staged data require adding USE_USER_STAGED_EXTRN_FILES: true to the config.yaml file

[MichaelLueken]

While running various WE2E tests on Jet and Cheyenne, it has been noted that not adding USE_USER_STAGED_EXTRN_FILES: true in the config.*.yaml file, the test will attempt to pull ICs and LBCs from /glade/p/ral/jntp/UFS_CAM/COMGFS/gfs.${yyyymmdd}/${hh} on Cheyenne (rather than /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data) and /public/data/grids/ on Jet (rather than /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data).

These non-EPIC maintained locations are found in the ush/machine/*.yaml files under:

data:
  ics_lbcs:

An example of this, on Cheyenne, can be seen in the following comment in PR #732 (comment). As can be seen, the test is looking for staged data in /glade/p/ral/jntp/UFS_CAM/COMGFS/gfs.20220810/06, not /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data.

Similarly, on Jet, attempting to run the nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km test, the machine attempted to pull data from /public/data/grids/gfs/0p25deg/grib2 rather than /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data. Adding USE_USER_STAGED_EXTRN_FILES: true to the config.nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km.yaml file allowed the test to successfully find the staged data on the machine.

One more example on Jet, from PR #757 (comment), the community test attempted to pull data from /public/data/grids/gfs/0p25deg/grib2, rather than /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data. The test then tried pulling data from HPSS, but since HPSS is down, the test failed to get ICs and LBCs for the test, resulting in the failure of the test. Manually running the test with USE_USER_STAGED_EXTRN_FILES: true set in config.community.yaml allowed the test to successfully find the staged data on the machine.

Should the WE2E test config.*.yaml files include USE_USER_STAGED_EXTRN_FILES: true or is this showing that there is an issue elsewhere in the workflow generation?

[MichaelLueken]

As discussed during today's SRW App Code Management meeting, the tests that appear to fail due to pointing to non-staged data locations are:

Cheyenne Intel:

nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16 - This test isn't currently being run (it is commented out in the coverage.cheyenne machine_suite due to missing staged data). There are no log files for this test in Jenkins. The test failed in the get_extrn_ics task with the following message:

INFO: Getting file: /glade/p/ral/jntp/UFS_CAM/COMGFS/gfs.20220810/06/gfs.t06z.sfcf006.nc

INFO: File does not exist on disk
 /glade/p/ral/jntp/UFS_CAM/COMGFS/gfs.20220810/06/gfs.t06z.sfcf006.nc
 try using: --input_file_path <your_path>

Ultimately, there are two issues with this specific test. First, it is attempting to pull data from /glade/p/ral/jntp/UFS_CAM/COMGFS/gfs.20220810/06. This is set in the ush/machine/cheyenne.yaml file as:

data:
  ics_lbcs:
    FV3GFS: /glade/p/ral/jntp/UFS_CAM/COMGFS/gfs.${yyyymmdd}/${hh}

It should be using the EPIC-maintained location - /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data, also set in the machine file.

The second issue is that the EPIC-maintained location - /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data, doesn't actually contain the necessary data for the test (there is no /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data/netcdf/2022081006).

Cheyenne GNU:

The Jenkins pipeline for these runs can be found - /glade/scratch/epicufsrt/jenkins/workspace/fs-srweather-app_pipeline_PR-728

nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km - this test passes, but it is pulling data from /glade/p/ral/jntp/UFS_CAM/COMGFS/gfs.20190615/00, rather than the EPIC-maintained location.
/glade/scratch/epicufsrt/jenkins/workspace/fs-srweather-app_pipeline_PR-728/expt_dirs/nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km

Gaea:

Both the:

data:
  ics_lbcs:
    FV3GFS:
      nemsio: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/FV3GFS/nemsio/${yyyymmdd}${hh}
      grib2: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/${yyyymmdd}${hh} 

and:

TEST_EXTRN_MDL_SOURCE_BASEDIR: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data

are identical in the ush/machine/gaea.yaml file.

Hera GNU:

The Jenkins pipeline for these runs can be found - /scratch1/NCEPDEV/stmp2/role.epic/jenkins/workspace/fs-srweather-app_pipeline_PR-728/

nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 - this test passes because Hera has access to HPSS, so can pull the data. The Jenkins logs for this test can be found - /scratch1/NCEPDEV/stmp2/role.epic/jenkins/workspace/fs-srweather-app_pipeline_PR-728/expt_dirs/nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16. Here, the test isn't even trying to copy the staged data from disk, but going directly to HPSS.

Jet:

Manually testing the nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km script on Jet (the test fails in the run_post task intermittently, so I test it on Hera and Jet instead), showed that the test failed while HPSS was down for maintenance due to attempting to pull data from:

data:
  ics_lbcs:
    FV3GFS:
      grib2: /public/data/grids/gfs/0p25deg/grib2

which is set in the ush/machine/jet.yaml file, rather than /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data, which is also set in the machine yaml file. For this manual test, setting USE_USER_STAGED_FILES: true allowed the test to run through without issue.

community - this test passes now the HPSS is back, but the failure is due to get_extrn_ics/lbcs attempting to copy data from /public/data/grids/gfs/0p25deg/grib2 rather than from /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data. The log files can be found - /lfs4/HFIP/hfv3gfs/role.epic/jenkins/workspace/fs-srweather-app_pipeline_PR-728/expt_dir/community. Setting USE_USER_STAGED_FILES: true and manually running corrected the issue.

Orion:

nco - this test passes because it is able to pull the necessary data from AWS. It doesn't look like the necessary data to run this has been staged on the machine. Please see /work/noaa/epic-ps/role-epic-ps/jenkins/workspace/fs-srweather-app_pipeline_PR-728/expt_dirs/nco for the Jenkins log files.

tldr (too long, didn't read):

The majority of the WE2E test config files do, in fact, contain USE_USER_STAGED_FILES: true. Those that don't are: community, nco, nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km, nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16, and nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16. Of these five tests,nco and nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16 don't have the necessary data in the EPIC-maintained locations currently, so adding this variable to these tests will lead to failures.

[gspetro-NOAA]

USE_USER_STAGED_EXTRN_FILES: true has been added to those WE2E tests that required it.