We added support for time-varying reference fields in Iris v3.13, in response to the adoption of time-varying orography in the MetOffice UM. #5369
Also in iris-grib, and relying on the above charge, we recently added support for hybrid pressure levels.
With this small fix, introduced along with the (reduced-gaussian grid saving)[https://github.com/SciTools/iris-grib/pull/813].
At least, what's new is, loading data with varying orography from UM files (FF/PP).
.. whereas this type of data has been around much longer with GRIB data sourced from ECMWF, as their model uses hybrid-pressure levels where the reference is a "surface_pressure" -- which, naturally, is always time-dependent.
However, until the above fix, Iris couldn't load that and create an appropriate hybrid air_pressure coordinate, which it now does
(-- hence, the iris-grib change was accepted without a "legacy mode" control, since it is seen as a "fix").
However .. some users are now reporting a serious performance hit on loading this type of data, compared to previous behaviour.
( Thanks to @lbdreyer for raising this. )
Notes:
I'm still investigating the context for this, but things which I think are relevant:
-
I think the problems are confined to file formats based on 2D "fields" -- i.e. notably FF, PP and GRIB.
In those cases, we generally get 2D fields which become "raw cubes" (as in iris.load_raw), which are then combined into larger cubes with dimensions of (at least) (T, Y, X) with a merge operation.
-
Where there is a varying reference (i.e. usually, a time variation), the "raw" cubes now have a length-1 "time" dimension.
- implemented here -- note: this only happens when it finds multiple reference fields (usually, for different timepoints)
- this is necessary because it indicates to a subsequent 'combine' operation that the reference coords map the time dimension
-- without which (i.e. as previously), Iris will refuse to combine these cubes as their 2D (Y, X) reference coords would have to be identical.
- so, the relevant raw-cube combination is then a concatenate rather than a merge, within the umbrella 'combine' operation.
From initial investigations, the code seems to be spending a lot (more) time in at least two key areas:
(1) in 'deferencing' the reference field for each raw cube, which also involves a possible transpose and regridding
From an initial look it seems that, although regrid results are cached, it's still costly just to pass through this code + create the extra coordinates.
(2) in the 'merge' stage of a subsequent concatenation operation (with the general 'combine' loading phase).
This is tricky because we need to compare every raw cube with all 'protocubes' to confirm that merging can happen, which means comparing all coords.
We added a control so we can choose to either realise references (the legacy mode) or leave them lazy -- but either case presents possible performance problems -- either purely on time-taken, or in heavy memory usage (which also quickly leads to time degradation via virtual memory and swap space)
Using 'lazy references' here causes a serious time hit, because the comparisons require fetching data from disk, and that must be done over + over.
NOTE: I think however that it may be possible to address this using hashing of comparisons in merge, as we already do for concatenate.
A simple, but drastic possible solution :
We could consider creating a control to "turn off" the creation of the hybrid vertical coordinates altogether.
Note: we can already disable multiple reference handling, via COMBINE_POLICY.support_multiple_references but that may not restore all the performance lost
We added support for time-varying reference fields in Iris v3.13, in response to the adoption of time-varying orography in the MetOffice UM. #5369
Also in iris-grib, and relying on the above charge, we recently added support for hybrid pressure levels.
With this small fix, introduced along with the (reduced-gaussian grid saving)[https://github.com/SciTools/iris-grib/pull/813].
At least, what's new is, loading data with varying orography from UM files (FF/PP).
.. whereas this type of data has been around much longer with GRIB data sourced from ECMWF, as their model uses hybrid-pressure levels where the reference is a "surface_pressure" -- which, naturally, is always time-dependent.
However, until the above fix, Iris couldn't load that and create an appropriate hybrid
air_pressurecoordinate, which it now does(-- hence, the iris-grib change was accepted without a "legacy mode" control, since it is seen as a "fix").
However .. some users are now reporting a serious performance hit on loading this type of data, compared to previous behaviour.
( Thanks to @lbdreyer for raising this. )
Notes:
I'm still investigating the context for this, but things which I think are relevant:
I think the problems are confined to file formats based on 2D "fields" -- i.e. notably FF, PP and GRIB.
In those cases, we generally get 2D fields which become "raw cubes" (as in
iris.load_raw), which are then combined into larger cubes with dimensions of (at least) (T, Y, X) with a merge operation.Where there is a varying reference (i.e. usually, a time variation), the "raw" cubes now have a length-1 "time" dimension.
-- without which (i.e. as previously), Iris will refuse to combine these cubes as their 2D (Y, X) reference coords would have to be identical.
From initial investigations, the code seems to be spending a lot (more) time in at least two key areas:
(1) in 'deferencing' the reference field for each raw cube, which also involves a possible transpose and regridding
From an initial look it seems that, although regrid results are cached, it's still costly just to pass through this code + create the extra coordinates.
(2) in the 'merge' stage of a subsequent concatenation operation (with the general 'combine' loading phase).
This is tricky because we need to compare every raw cube with all 'protocubes' to confirm that merging can happen, which means comparing all coords.
We added a control so we can choose to either realise references (the legacy mode) or leave them lazy -- but either case presents possible performance problems -- either purely on time-taken, or in heavy memory usage (which also quickly leads to time degradation via virtual memory and swap space)
Using 'lazy references' here causes a serious time hit, because the comparisons require fetching data from disk, and that must be done over + over.
NOTE: I think however that it may be possible to address this using hashing of comparisons in merge, as we already do for concatenate.
A simple, but drastic possible solution :
We could consider creating a control to "turn off" the creation of the hybrid vertical coordinates altogether.
Note: we can already disable multiple reference handling, via
COMBINE_POLICY.support_multiple_referencesbut that may not restore all the performance lost