fates parameter file auto-build for all tests #2336

rgknox · 2024-01-25T15:31:07Z

Description of changes

This enables the automatic building of the fates parameter file binary for all tests. This calls ncgen from the shell_commands script in the Fates/ testdef folder, to operate on the fates default file that is version controlled.

Specific notes

This implementation is incomplete. In order to get this to work for all tests, I had to place the newly built binaries in the a new folder in the fates source tree. The reason for this is because some of the tests are multi-phase (PEM, ERP, etc). Each of these phases needs access to either the same parameter file, or an exact copy of it. However, the shell_commands (as far as my test show) script is only called the first time, so both parts of the test need access to the same file. Unfortunately, the xml files in both parts of the tests, do not provide any file-paths that are common to phase (I looked pretty thoroughly but maybe missed something), located somewhere on the scratch partition. For instance, they both have different cases, which makes it tough for us to locate the parameter file on the second test, if it has a different case as the first. I also tried using CIME_OUTPUT_ROOT, the sharedlib build location.

There is an xml entry in env_test.xml that is TEST_ARGV. This holds the root folder for the current test environment, and the id of the specific test currently run. With these two bits of information we could place a binary file that is accessible to all phases of a test. However, this information does not seem to be available via xml query at the time we run the shell_commands script.

Another location that might be better than the source, at least for the time being would be to put all these files in the CIME_OUTPUT_ROOT, which is usually just the scratch folder where all cases and tests go. Each parameter file could have the test name and hash in it, to prevent redundancy. The downside is that the root scratch folder starts to fill up.

Contributors other than yourself, if any:

@ekluzek @glemieux @adrifoster

Are answers expected to change (and if so in what way)?

Any User Interface Changes (namelist or namelist defaults changes)?

Testing performed, if any:

rgknox · 2024-01-25T16:06:18Z

Here is a list of the parameter test file binaries it generates, one fates test creates 5.9M of data:

~/ctsm/src/fates/parameter_files> ls binaries/
ERP_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold.0124-200432de_int-params.nc
ERP_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold.20240124_191528_ohyuz4-params.nc.text
ERP_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold.20240124_193525_2d6yfg-params.nc
ERP_D_P128x2_Ld3.f19_g17.I2000Clm50FatesCru.derecho_intel.clm-FatesCold.0124-200432de_int-params.nc
ERP_Ld3.f09_g17.I2000Clm50FatesRs.derecho_intel.clm-FatesCold.0124-200432de_int-params.nc
ERP_P256x2_Ld30.f45_f45_mg37.I2000Clm51FatesRs.derecho_intel.clm-mimicsFatesCold.0124-200432de_int-params.nc
ERS_D_Ld15.5x5_amazon.I2000Clm50FatesRs.derecho_gnu.clm-FatesColdSeedDisp.0124-200432de_gnu-params.nc
ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.derecho_gnu.clm-FatesColdTwoStreamNoCompFixedBioGeo.0124-200432de_gnu-params.nc
ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdTreeDamage.0124-200432de_int-params.nc
ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdTwoStream.0124-200432de_int-params.nc
ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLandUse.0124-200432de_int-params.nc
ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLUH2.0124-200432de_int-params.nc
ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPRT2.0124-200432de_int-params.nc
ERS_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesCold.0124-200432de_gnu-params.nc
ERS_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold.0124-200432de_int-params.nc
ERS_D_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdHydro.0124-200432de_int-params.nc
ERS_D_Ld5.f10_f10_mg37.I2000Clm50Fates.derecho_intel.clm-FatesCold.0124-200432de_int-params.nc
ERS_D_Mmpi-serial_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesCold.0124-200432de_gnu-params.nc
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdFixedBiogeo.0124-200432de_int-params.nc
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoComp.0124-200432de_int-params.nc
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoCompFixedBioGeo.0124-200432de_int-params.nc
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdSizeAgeMort.0124-200432de_int-params.nc
ERS_Ld5.f19_g17.I2000Clm45Fates.derecho_intel.clm-FatesCold.0124-200432de_int-params.nc
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-Fates.0124-200432de_int-params.nc
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLogging.0124-200432de_int-params.nc
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoFire.0124-200432de_int-params.nc
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPPhys.0124-200432de_int-params.nc
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdST3.0124-200432de_int-params.nc
ERS_Ld9.f10_f10_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdCH4Off.0124-200432de_int-params.nc
ERS_Lm13.f10_f10_mg37.I2000Clm50Fates.derecho_gnu.clm-FatesCold.0124-200432de_gnu-params.nc
ERS_Lm13.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdNoComp.0124-200432de_int-params.nc
ERS_P128x1_Lm25.f10_f10_mg37.I2000Clm51Fates.derecho_intel.clm-FatesColdNoComp.0124-200432de_int-params.nc
PEM_D_Ld15.5x5_amazon.I2000Clm50FatesRs.derecho_gnu.clm-FatesColdSeedDisp.0124-200432de_gnu-params.nc
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm51Fates.derecho_gnu.clm-FatesPRISM--clm-NEON-FATES-YELL.0124-200432de_gnu-params.nc
SMS_Lm13.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesCold.0124-200432de_gnu-params.nc
SMS_Lm13.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold.0124-200432de_int-params.nc
SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesColdHydro.0124-200432de_gnu-params.nc
SMS_Lm6.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-Fates.0124-200432de_int-params.nc

samsrabin · 2024-01-25T16:38:01Z

This should probably happen in a SystemTest, not in shell_commands. See #2335.

samsrabin · 2024-01-25T16:49:25Z

See also discussion from CTSM SE meeting here.

I.e., PR ESCOMP#2326.

samsrabin · 2024-02-12T19:26:24Z

I'm unable to get this to work on Izumi. It might be something wrong with my environment. Could someone else try this and see?

./run_sys_tests --skip-compare --skip-generate -t ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream

ekluzek

@rgknox this is a nice second step to help us head towards #2126.

I have some questions that I'm wondering about that I added to the code changes. I'm not sure that should hold anything up. But, I marked it as request changes until we decide on this...

cime_config/testdefs/testmods_dirs/clm/Fates/shell_commands

cime_config/testdefs/testmods_dirs/clm/FatesColdSeedDisp/shell_commands

cime_config/testdefs/testmods_dirs/clm/Fates/shell_commands

samsrabin

Just a few suggestions in the code. Also:

Ensure this works for various users on Izumi.

cime_config/testdefs/testmods_dirs/clm/FatesColdSeedDisp/shell_commands

cime_config/testdefs/testmods_dirs/clm/Fates/shell_commands

glemieux · 2024-02-15T18:11:29Z

I'm unable to get this to work on Izumi. It might be something wrong with my environment. Could someone else try this and see?
./run_sys_tests --skip-compare --skip-generate -t ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream

I'm seeing a failure trying this as well with the following:

RUN: /scratch/cluster/glemieux/tests_0215-110541iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0215-110541iz/shell_commands
FROM: /scratch/cluster/glemieux/tests_0215-110541iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0215-110541iz
  stat: 1

  errput: ncgen: No such file or directory
        (../../ncgen/genbin.c:58)
Traceback (most recent call last):
  File "/home/glemieux/ctsm/src/fates/tools/modify_fates_paramfile.py", line 355, in <module>
    main()
  File "/home/glemieux/ctsm/src/fates/tools/modify_fates_paramfile.py", line 95, in main
    shutil.copyfile(args.inputfname, tempfilename)
  File "/cluster/anaconda-23.11.0/lib/python3.11/shutil.py", line 256, in copyfile
    with open(src, 'rb') as fsrc:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/glemieux/ctsm/src/fates/parameter_files/binaries/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0215-110541iz-params.nc'
Leaving broken case dir /scratch/cluster/glemieux/tests_0215-110541iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.0215-110541iz

Looking at the fates directory structure, the binaries directory isn't getting built ~~for some reason~~ because ncgen isn't found. @samsrabin is this the same error you were seeing?

samsrabin · 2024-02-15T18:28:36Z

That is indeed the same error I was seeing, but I don't think it's ncgen not being found. I think there might be a problem with the ncgen installation.

…ests -f.

samsrabin · 2024-02-15T19:33:17Z

I'm actually getting a similar error on Derecho, too. From SMS_D_Ld3.f10_f10_mg37.I2000Clm50FatesRs.derecho_intel:

Adding user mods directory /glade/u/home/samrabin/ctsm_fates-auto-params/cime_config/testdefs/testmods_dirs/clm/Fates
RUN: /glade/derecho/scratch/samrabin/tests_0215-114155de/SMS_D_Ld3.f10_f10_mg37.I2000Clm50FatesRs.derecho_intel.clm-Fates.0215-114155de/shell_commands
FROM: /glade/derecho/scratch/samrabin/tests_0215-114155de/SMS_D_Ld3.f10_f10_mg37.I2000Clm50FatesRs.derecho_intel.clm-Fates.0215-114155de
  stat: 1

  errput: ncgen: No such file or directory
	(/home/conda/feedstock_root/build_artifacts/libnetcdf_1650908392318/work/ncgen/genbin.c:genbin_netcdf:63)

Tests in /glade/derecho/scratch/samrabin/tests_0215-114155de/.

samsrabin · 2024-02-15T19:40:47Z

The issue is that ${SRCROOT}/src/fates/parameter_files/binaries/ doesn't exist. Will fix in my PR.

ekluzek · 2024-02-15T19:43:56Z

@samsrabin thanks for the work on this. Note, that fates/parameter_files is under the FATES external so adding a binary_files subdirectory would require a PR to FATES. And I think for git, you have to have at least a README file in the directory for it to show up when you check it out...

samsrabin · 2024-02-15T19:47:46Z

@ekluzek Good point; adding this directory will make the FATES checkout unclean. @rgknox I think you need to make a new FATES tag that has an empty parameter_files/binaries/ directory, then update Externals.cfg here to point to that. No; see below.

samsrabin · 2024-02-15T19:56:09Z

Wait, @ekluzek, even if the new directory is canonically in FATES, won't the checkout be unclean once the parameter file is generated?

samsrabin · 2024-02-15T19:58:24Z

Actually… the checkout looks clean. manage_externals/checkout_externals -S gives no warning, and git status in src/fates is clean, even with the new directory and parameter files generated. This might be because .nc files are ignored by src/fates/.gitignore.

ekluzek · 2024-02-15T20:07:56Z

@samsrabin yes exactly. But, it's good that you showed that's the case. It's good to confirm.

rgknox · 2024-02-15T21:48:35Z

That directory needs to be added in fates, sorry for that, I'll get it added to the next FATES PR.
UPDATE: Sam added a mkdir -p call to the scripting, so we don't need this directory added anymore.

Improvements and fix for fates-auto-params

samsrabin · 2024-02-22T17:18:23Z

That merge commit I just did was to resolve conflicts introduced in my PR. They were only related to run_sys_tests.py and its testing. They're now resolved, and make all in python/ is still clean.

rgknox · 2024-02-24T14:35:58Z

clm_aux on derecho, ok with exception:

~~FAIL SMS.f10_f10_mg37.I2000Clm50BgcCrop.derecho_nvhpc.clm-crop MODEL_BUILD time=212~~ (THIS TEST PASSES AFTER RESUBMITTING)

FAIL DAE_C2_D_Lh12.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-DA_multidrv RUN time=304

ERROR: ERROR: Unrecognized line ('/bin/bash: module: line 1: syntax error: unexpected end of file

samsrabin · 2024-02-24T20:56:52Z

That dang DAE test! Try resubmitting it.

ekluzek · 2024-02-26T23:24:42Z

I tried the izumi test that was failing before and it works for me, so I checked that item off, which puts this in a ready to merge mode.

rgknox · 2024-02-27T02:52:42Z

This test fails create case: ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream

RUN: /scratch/cluster/rgknox/tests_0226-121631iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.GC.0226-121631iz_nag/shell_commands
FROM: /scratch/cluster/rgknox/tests_0226-121631iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.GC.0226-121631iz_nag
  stat: 1

  errput: Traceback (most recent call last):
  File "/home/rgknox/ctsm/src/fates/tools/modify_fates_paramfile.py", line 35, in <module>
    from scipy.io import netcdf as nc
ImportError: No module named scipy.io
Leaving broken case dir /scratch/cluster/rgknox/tests_0226-121631iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.GC.0226-121631iz_nag
ERROR: Command: '/scratch/cluster/rgknox/tests_0226-121631iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.GC.0226-121631iz_nag/shell_commands' failed with error 'Traceback (most recent call last):
  File "/home/rgknox/ctsm/src/fates/tools/modify_fates_paramfile.py", line 35, in <module>
    from scipy.io import netcdf as nc
ImportError: No module named scipy.io' from dir '/scratch/cluster/rgknox/tests_0226-121631iz/ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.GC.0226-121631iz_nag'

 ---------------------------------------------------
2024-02-26 13:08:10: CREATE_NEWCASE FAILED for test 'ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream'.

However, i'm able to load scipy manually when I run python. Also, this test passed when I ran it stand-alone.. ie, this test passed:

./create_test ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream

rgknox · 2024-02-27T02:53:00Z

DAE_C2_D_Lh12.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-DA_multidrv also still fails after re-submitting

ekluzek · 2024-02-28T17:47:02Z

Since this isn't critical to come in now, we will plan on delaying this to fix the conda env issue on izumi (I think #2385 will fix this). @samsrabin also has some analysis that shows that there is a race condition for the DAE test that sometime results in a file being gzipped before something else needs I don't think the DAE issue should hold this one up, but that is another good thing to have come in.

Undoing approval until it's actually ready

glemieux · 2024-08-26T19:42:36Z

Getting this prioritized came up in discussion around NGEET/fates#1236. @ckoven suggested that we may want to also move towards building from the xml patch file.

wwieder · 2024-10-31T16:28:00Z

@ekluzek will start a document on this and shared it with interested parties by our Nov 6 meeting.

adding fates parameter auto-build to all tests

3179627

samsrabin added 2 commits February 12, 2024 10:56

Bring in changes to run_sys_tests from ctsm5.1.dev166.

5404a7c

I.e., PR ESCOMP#2326.

run_sys_tests now checks modules needed for FatesColdSeedDisp.

5e5d485

ekluzek requested changes Feb 15, 2024

View reviewed changes

cime_config/testdefs/testmods_dirs/clm/Fates/shell_commands Outdated Show resolved Hide resolved

cime_config/testdefs/testmods_dirs/clm/FatesColdSeedDisp/shell_commands Outdated Show resolved Hide resolved

cime_config/testdefs/testmods_dirs/clm/Fates/shell_commands Show resolved Hide resolved

ekluzek assigned rgknox and ekluzek Feb 15, 2024

ekluzek added FATES A change needed for FATES that doesn't require a FATES API update. testing additions or changes to tests labels Feb 15, 2024

samsrabin requested changes Feb 15, 2024

View reviewed changes

cime_config/testdefs/testmods_dirs/clm/FatesColdSeedDisp/shell_commands Outdated Show resolved Hide resolved

cime_config/testdefs/testmods_dirs/clm/Fates/shell_commands Outdated Show resolved Hide resolved

Remove code duplication in Fates*/shell_commands.

fe5d12a

Fix check of modify_fates_paramfile module availability for run_sys_t…

ec6cd21

…ests -f.

Make dir for FATESPARAMFILE if needed.

d82c7b5

samsrabin mentioned this pull request Feb 15, 2024

Improvements and fix for fates-auto-params rgknox/ctsm#5

Merged

rgknox and others added 2 commits February 22, 2024 11:29

Merge pull request #5 from samsrabin/fates-auto-params-ssr

c1bd84d

Improvements and fix for fates-auto-params

Merge branch 'master' into fates-auto-params

ed41f90

samsrabin previously approved these changes Feb 22, 2024

View reviewed changes

added python to the call on modify_fates_paramfile.py, needed for izumi

267fe14

ekluzek previously approved these changes Feb 26, 2024

View reviewed changes

ekluzek mentioned this pull request Feb 26, 2024

Have the FATES parameter file created at runtime in the buildnml #2126

Open

glemieux mentioned this pull request Jun 10, 2024

Switching the default fates_cnp_prescribed_* parameters from 1 to 0 NGEET/fates#1211

Closed

samsrabin requested review from samsrabin and ekluzek June 20, 2024 16:54

samsrabin added the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Aug 26, 2024

wwieder removed the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fates parameter file auto-build for all tests #2336

fates parameter file auto-build for all tests #2336

rgknox commented Jan 25, 2024 •

edited by ekluzek

Loading

rgknox commented Jan 25, 2024

samsrabin commented Jan 25, 2024

samsrabin commented Jan 25, 2024

samsrabin commented Feb 12, 2024

ekluzek left a comment

samsrabin left a comment •

edited by ekluzek

Loading

glemieux commented Feb 15, 2024 •

edited

Loading

samsrabin commented Feb 15, 2024

samsrabin commented Feb 15, 2024

samsrabin commented Feb 15, 2024

ekluzek commented Feb 15, 2024

samsrabin commented Feb 15, 2024 •

edited

Loading

samsrabin commented Feb 15, 2024

samsrabin commented Feb 15, 2024

ekluzek commented Feb 15, 2024

rgknox commented Feb 15, 2024 •

edited

Loading

samsrabin commented Feb 22, 2024

rgknox commented Feb 24, 2024 •

edited

Loading

samsrabin commented Feb 24, 2024

ekluzek commented Feb 26, 2024

rgknox commented Feb 27, 2024

rgknox commented Feb 27, 2024

ekluzek commented Feb 28, 2024

glemieux commented Aug 26, 2024

wwieder commented Oct 31, 2024

fates parameter file auto-build for all tests #2336

Are you sure you want to change the base?

fates parameter file auto-build for all tests #2336

Conversation

rgknox commented Jan 25, 2024 • edited by ekluzek Loading

Description of changes

Specific notes

rgknox commented Jan 25, 2024

samsrabin commented Jan 25, 2024

samsrabin commented Jan 25, 2024

samsrabin commented Feb 12, 2024

ekluzek left a comment

Choose a reason for hiding this comment

samsrabin left a comment • edited by ekluzek Loading

Choose a reason for hiding this comment

glemieux commented Feb 15, 2024 • edited Loading

samsrabin commented Feb 15, 2024

samsrabin commented Feb 15, 2024

samsrabin commented Feb 15, 2024

ekluzek commented Feb 15, 2024

samsrabin commented Feb 15, 2024 • edited Loading

samsrabin commented Feb 15, 2024

samsrabin commented Feb 15, 2024

ekluzek commented Feb 15, 2024

rgknox commented Feb 15, 2024 • edited Loading

samsrabin commented Feb 22, 2024

rgknox commented Feb 24, 2024 • edited Loading

samsrabin commented Feb 24, 2024

ekluzek commented Feb 26, 2024

rgknox commented Feb 27, 2024

rgknox commented Feb 27, 2024

ekluzek commented Feb 28, 2024

glemieux commented Aug 26, 2024

wwieder commented Oct 31, 2024

rgknox commented Jan 25, 2024 •

edited by ekluzek

Loading

samsrabin left a comment •

edited by ekluzek

Loading

glemieux commented Feb 15, 2024 •

edited

Loading

samsrabin commented Feb 15, 2024 •

edited

Loading

rgknox commented Feb 15, 2024 •

edited

Loading

rgknox commented Feb 24, 2024 •

edited

Loading