ISM Compliance checker: treat "out of range" as warnings not errors

[aaschwanden]

For several variables we see out of range errors like

NAMING Tests
 - Filename year range 1985-2014 is well formed: OK
 - Variable 'xvelsurf' from the file name is present in the file: OK
 - No unexpected variables in the file: OK
 - Variable 'xvelsurf' dimensions (time, y, x) match the requested x,y,t: OK
** Tested Variable: xvelsurf

NUMERICAL Tests
 - The unit is correct: m s^-1 (equivalent to the requested m s-1)
 - Missing values: OK
 - ERROR: The minimum value (-0.0014995316741988063) is out of range. Min value accepted: -0.0008
 - ERROR: The maximum value (0.0015717364149168134) is out of range. Max value accepted: 0.0008
SPATIAL Tests
 - Grid: Lowest left corner is well defined.
 - Grid: Upper right corner is well defined.

as the min/max accepted bounds are somewhat arbitrary, we suggest treating those as warnings not errors. The SSA solver can produce localized grid points with large velocities that are not indicative of something wrong with the simulation per se. Which means these errors can't be fixed on our end.

[xylar]

Thanks @aaschwanden. Agreed that the velocity bounds are the "yellow flag" kind rather than hard limits. The checker already has a per-variable range_severity column in the data request for exactly this, and every row was still set to error.

I've opened ismip/ISM_SimulationChecker#31 to switch the eight velocity components (xvelsurf, yvelsurf, zvelsurf, xvelbase, yvelbase, zvelbase, xvelmean, yvelmean) to warning. The out-of-range value is still reported so it can be looked at, but it no longer fails the file. The other bounds (fractions, thicknesses, temperatures, fluxes) stay as errors since they look like hard physical limits to me, but if you are seeing spurious failures on any of those, say which and we can move them too.

Once that's merged, we'll tag 0.3.0 so the fix reaches conda-forge.

— Claude Code (writing on behalf of @xylar)

[aaschwanden]

Thank you Xylar:

I would extent this to other variables too:

For example, negative heat flux is physical and (in our case) can occur in recently de-glaciated areas were suddenly bedrock is exposed:

NAMING Tests
 - Filename year range 1985-2014 is well formed: OK
 - Variable 'hfgeoubed' from the file name is present in the file: OK
 - No unexpected variables in the file: OK
 - Variable 'hfgeoubed' dimensions (time, y, x) match the requested x,y,t: OK
** Tested Variable: hfgeoubed

NUMERICAL Tests
 - The unit is correct: W m^-2 (equivalent to the requested W m-2)
 - Missing values: OK
 - ERROR: The minimum value (-3.2648754119873047) is out of range. Min value accepted: 0.0
 - ERROR: The maximum value (2.143779993057251) is out of range. Max value accepted: 0.3```
 
 Similar for large fluxes, they may occur at the beginning of a simulation when a change in calving or so triggers a large event:
 
 ```Experiment: historical - File: libmassbfgr_GrIS_UAF_PISM_m001_CESM2-WACCM_f001_historical_C001_1985-2014.nc

NAMING Tests
 - Filename year range 1985-2014 is well formed: OK
 - Variable 'libmassbfgr' from the file name is present in the file: OK
 - No unexpected variables in the file: OK
 - Variable 'libmassbfgr' dimensions (time, y, x) match the requested x,y,t: OK
** Tested Variable: libmassbfgr

NUMERICAL Tests
 - The unit is correct: kg m^-2 s^-1 (equivalent to the requested kg m-2 s-1)
 - Missing values: OK
 - ERROR: The minimum value (-0.005117337219417095) is out of range. Min value accepted: -0.0003
 - The maximum value successfully verified.
SPATIAL Tests
 - Grid: Lowest left corner is well defined.
 - Grid: Upper right corner is well defined.
 - The grid resolution (1 km) was successfully verified.
CONSISTENCY Tests
 - WARNING: variable 'libmassbfgr' is missing in 14494 of 145288830 value(s) (0.00998%) where 'sftgrf' is greater than 0, so there is ice the file says nothing about. 5436 of them have 'sftgrf' below 0.01, which is the partly glaciated margin rather than the interior.
TIME Tests
 - Time is a record (unlimited) dimension: OK
 - Time axis: 30 annual FL time step(s) covering nominal years 1985-2014, as experiment 'historical' requires: OK
ATTRIBUTE Tests
 - Global attributes: OK
 - Coordinate attributes: OK
 - Variable attributes: OK
 - Fill value attributes: OK
 - Dtype attributes: OK
 - Packing attributes: OK```
 
 So I would treat all out of bounds errors as warnings. Flagging them as errors poses the risk of accidentally discarding valid simulations during analysis.

[xylar]

Thanks @aaschwanden, those two examples make the point well: the check compares the field's extreme values against the bounds, and the min/max of a 145-million-value field over 30 years is the least robust statistic there is. Any single cell can trip it, and classifying variable by variable was not going to be the last round.

We're hesitant to make every bound a warning outright, though, because the range check is the only thing that catches a units attribute that lies: a velocity file in m/yr whose attribute says m s-1 passes the units check (which reads the attribute, not the data), and only the range check notices that every cell is 3×10⁷ too large. As a warning, that would get ignored, and a file like that in the archive is worse than a false error.

So ismip/ISM_SimulationChecker#32 grades the finding by how much of the field is outside the bounds instead:

• fewer than 1% of the values that hold a number → warning
• 1% or more → error

Either way the message now says the count and share, e.g. 14 of 145288830 value(s) (0.00001%) are below it, so the reader can judge. Your three cases (velocity outliers, negative heat flux in freshly deglaciated cells, one calving event) would all be warnings; a whole field in the wrong units stays an error. The per-variable range_severity column stays as an escape hatch for a bound that turns out to be soft everywhere, not just at a few cells.

@hgoelzer is out so I'll go ahead and merge the PR and make a release of the code. After that, if you have a chance to run it against the UAF files in the meantime, it would be good to know whether any of them comes anywhere near the 1% line.

— Claude Code (writing on behalf of @xylar)

[tkleiner]

I ran into similar issues for 'strbasemag', 'acbf` and 'libmassbffl', and would prefer to have out-of-range values treated as warnings.

[xylar]

@aaschwanden @tkleiner, Please try isschecker 0.4.0, which should be available on conda-forge.

[aaschwanden]

awesome, thank you @xylar

Now we only have one bound error left

NAMING Tests
 - Filename year range 1985-2014 is well formed: OK
 - Variable 'hfgeoubed' from the file name is present in the file: OK
 - No unexpected variables in the file: OK
 - Variable 'hfgeoubed' dimensions (time, y, x) match the requested x,y,t: OK
** Tested Variable: hfgeoubed

NUMERICAL Tests
 - The unit is correct: W m^-2 (equivalent to the requested W m-2)
 - Missing values: OK
 - ERROR: The minimum value (-3.2648754119873047) is out of range. Min value accepted: 0.0; 9642912 of 145288830 value(s) (6.64%) are below it.
 - WARNING: The maximum value (2.143779993057251) is out of range. Max value accepted: 0.3; 124199 of 145288830 value(s) (0.0855%) are above it.

and three over-eager flaggings of correct UDUNITS

NAMING Tests
 - Filename year range 1985-2014 is well formed: OK
 - Variable 'ligroundf' from the file name is present in the file: OK
 - No unexpected variables in the file: OK
 - Variable 'ligroundf' dimensions (time, y, x) match the requested x,y,t: OK
** Tested Variable: ligroundf

NUMERICAL Tests
 - ERROR: The unit of the variable is kg m^-2 second^-1 and should be kg m-2 s-1
 - Missing values: OK
 - The minimum value successfully verified.
 - The maximum value successfully verified.

and kelvin

NAMING Tests
- Filename year range 1985-2014 is well formed: OK
- Variable 'litempbotgr' from the file name is present in the file: OK
- No unexpected variables in the file: OK
- Variable 'litempbotgr' dimensions (time, y, x) match the requested x,y,t: OK
** Tested Variable: litempbotgr

NUMERICAL Tests
- ERROR: The unit of the variable is kelvin and should be K
- Missing values: OK
- The minimum value successfully verified.
- The maximum value successfully verified.

and here

Experiment: historical - File: litemptop_GrIS_UAF_PISM_m001_CESM2-WACCM_f001_historical_C001_1985-2014.nc

NAMING Tests
 - Filename year range 1985-2014 is well formed: OK
 - Variable 'litemptop' from the file name is present in the file: OK
 - No unexpected variables in the file: OK
 - Variable 'litemptop' dimensions (time, y, x) match the requested x,y,t: OK
** Tested Variable: litemptop

NUMERICAL Tests
 - ERROR: The unit of the variable is kelvin and should be K
 - Missing values: OK

[xylar]

Thanks @aaschwanden, both are fixed on main now:

• hfgeoubed: Make the hfgeoubed range a warning however much of the field is outside ISM_SimulationChecker#33 sets its range_severity to warning, so it never fails a file however much of the field is outside the bounds. 6.6% of a 1 km Greenland grid is about the ice-free land area, so it looks like PISM reports the flux at the top of its bedrock thermal layer over all land, and over bare ground under a warming surface that flux points down. The outside_domain fill policy allows the file to cover that, and a single-file check cannot tell ice-free land from ice, so the bound (which describes the flux under ice) is soft for this variable.

• kelvin and second: Compare units through UDUNITS instead of a parser of our own ISM_SimulationChecker#34 replaces the checker's own units parser with UDUNITS-2 (via cf-units), so any spelling UDUNITS recognizes is accepted, names as well as symbols. m yr-1 is still not m s-1 and degC is still not K, so a unit at the wrong scale is caught as before.

A 0.5.0 release with both should be available on conda-forge soon. If anything else turns up when you run it, please let us know here.

— Claude Code (writing on behalf of @xylar)