ED runs die in PIO while trying to read params file, starting with cime3

[billsacks]

Bill Sacks < sacks@ucar.edu > - 2015-09-29 15:40:54 -0600
Bugzilla Id: 2221
Bugzilla CC: andre@ucar.edu, erik@ucar.edu, jedwards@ucar.edu, mvertens@ucar.edu, rfisher@ucar.edu,

With the updates to cime3 (planned for clm4_5_3_r131), these two ED runs now die while trying to read the params file:

SMS_Ld5.f10_f10.ICLM45ED.yellowstone_pgi.clm-edTest
SMS_Ld5.f19_g16.ICLM45ED.yellowstone_intel.clm-edTest

The deaths look like:

1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/lnd/clm2/paramdata/clm_params_ed.c150317.nc
1: 0
[then many instances of the following:]
9: pio_support::pio_die:: myrank= -1 : ERROR: pionfget_mod.F90:
9: 426 : Input/Output data amount mismatch
41:Image PC Routine Line Source
41:cesm.exe 000000000144CC38 Unknown Unknown Unknown
41:cesm.exe 0000000000D705D1 pio_support_mp_pi 120 pio_support.F90
41:cesm.exe 0000000000D6E645 pio_utils_mp_chec 59 pio_utils.F90
41:cesm.exe 0000000000E27DDD pionfget_mod_mp_g 426 pionfget_mod.F90.in
41:cesm.exe 00000000005A859D ncdio_pio_mp_ncd_ 1202 ncdio_pio.F90.in
41:cesm.exe 0000000000AFBBB2 paramutilmod_mp_r 50 paramUtilMod.F90
41:cesm.exe 00000000007BC556 edparamsmod_mp_ed 106 EDParamsMod.F90
41:cesm.exe 00000000005CE868 readparamsmod_mp_ 69 readParamsMod.F90
41:cesm.exe 0000000000508628 clm_initializemod 355 clm_initializeMod.F90
41:cesm.exe 00000000004F7DC8 lnd_comp_mct_mp_l 232 lnd_comp_mct.F90
41:cesm.exe 00000000004262AB component_mod_mp_ 229 component_mod.F90
41:cesm.exe 0000000000415D38 cesm_comp_mod_mp_ 1086 cesm_comp_mod.F90
41:cesm.exe 00000000004213ED MAIN__ 92 cesm_driver.F90

I wonder if this is showing up for the first time now because one of the changes in the cime3 update was:

NEW: pio_typename = 'pnetcdf'
BASELINE: pio_typename = 'netcdf'

i.e., could the use of pnetcdf be revealing a problem that until now went unnoticed?

[billsacks]

Erik Kluzek < erik@ucar.edu > - 2015-09-29 15:50:58 -0600

Bill, if you change it by hand for these cases to "netcdf" do the test cases work?

[billsacks]

Mariana Vertenstein < mvertens@ucar.edu > - 2015-09-29 15:55:16 -0600

(In reply to Erik Kluzek from comment #1)

Bill, if you change it by hand for these cases to "netcdf" do the test cases
work?

That is a good question. Right now all yellowstone configuration (other than serial) are using pnetcdf rather than netcdf. So that might be an issue.

[billsacks]

Bill Sacks < sacks@ucar.edu > - 2015-09-30 04:42:02 -0600

This one dies the same way:

RUN SMS_Ld5.f10_f10.ICLM45ED.yellowstone_intel.clm-edTest

[billsacks]

Bill Sacks < sacks@ucar.edu > - 2015-09-30 04:43:20 -0600

In response to Erik: I haven't had a chance to look into whether using netcdf fixes this problem. If it does, I'm not sure what it means. Note that I suspect that this has a different underlying cause than bug 1730 (this is a problem in a read, rather than a write).

[billsacks]

@ekluzek @bandre-ucar @rosiealice @jedwards4b @mvertens I'm closing this old issue: As far as I know it is no longer a problem.