error getting size of a grid

[karlmsmith]

Reported by @AnsleyManke on 7 Oct 2010 01:09 UTC
Define a large 4D grid. On LOADing the variable, we get an error that indicates the size of the grid couldn't be computed.

yes? use levitus_climatology
yes? define axis/t=1-jan-2000:1-jan-2005:1/units=days  longtime
yes? let ttemp = temp  + 0*t[gt=longtime]

yes? sh grid ttemp
    GRID (G002)
 name       axis              # pts   start                end
 XAXLEVITR LONGITUDE          360mr   20.5E                19.5E(379.5)
 YAXLEVITR LATITUDE           180 r   89.5S                89.5N
 ZAXLEVITR DEPTH (m)           20 i-  0                    5000
 LONGTIME  TIME              1828 r   01-JAN-2000 00:00    01-JAN-2005 00:00

yes? load ttemp
 **ERROR: request exceeds memory setting: A negative number of words were requested.
 *** NOTE: The current grid is most likely too large

However, the same variable on an even longer time axis does result in a size being computed.

! Continue, increasing the length of the axis from 5 to 10 years.

yes?  define axis/t=1-jan-2000:1-jan-2010:1/units=days  longtime
Replacing definition of axis LONGTIME
yes? show grid ttemp
yes? load ttemp
 **ERROR: request exceeds memory setting: 440616704 words were requested.
 *** NOTE: You can use SET MEMORY/SIZE=xxx to increase memory.
 *** NOTE: The "Memory use" section of the FERRET Users Guide has further tips.

So something's amiss in getting the grid size.

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/1758

[karlmsmith]

Comment by @AnsleyManke on 30 Nov 2010 21:16 UTC
Note that this bug goes back to older versions of Ferret.

In fact, the size of the variable using the longer time axis is returning an incorrect result too. It just happens that it represents an overflow that shows as a positive number.

The grid in the example above, for the 5-year axis, is 36018020*1828 = 2.1474836E+09, larger than what a 32-bit integer variable can represent. (Max signed integer is 2147483647).

For the 10-year axis,

yes? define axis/t=1-jan-2000:1-jan-2010:1/units=days longtime
yes? use levitus_climatology
yes? let ttemp = temp  + 0*t[gt=longtime]

yes? say `ttemp,return=size`
 !-> MESSAGE/CONTINUE 440616704
440616704

yes? say `ttemp[L=1:1600],return=size`
 !-> MESSAGE/CONTINUE 2073600000
2073600000

yes? say `ttemp[L=1:2000],return=size`
 !-> MESSAGE/CONTINUE -1702967296
-1702967296

There does not seem to be a flag for gnu gfortran to trap integer overflow. (There is for floating-point overflow; -ffpe-trap=overflow, but that is floating point only.)

Seems to me the solution is to do these calculations using floating-point, or to check for integer overflow using floating-point variables, when computing grid size. See mem/cgrid_size.F, cgrid_size_delta.F, cgrid_size_max.F.

[karlmsmith]

Comment by @AnsleyManke on 4 Sep 2014 19:09 UTC
Ferret memory is managed in blocks of memory, so the tests for whether we have enough memory already or need more is not the issue, only the computation of the size of the grid we're trying to load. This means that the size of the grid or other memory request can be computed as a real number, then the number of blocks computed from that, and everything is accurate.

The example above now looks like this:

yes? use levitus_climatology
yes? define axis/t=1-jan-2000:1-jan-2005:1/units=days  longtime
yes? let ttemp = temp  + 0*t[gt=longtime]

yes? sh grid ttemp
    GRID (G002)
 name       axis              # pts   start                end
 XAXLEVITR LONGITUDE          360mr   20.5E                19.5E(379.5)
 YAXLEVITR LATITUDE           180 r   89.5S                89.5N
 ZAXLEVITR DEPTH (m)           20 i-  0                    5000
 LONGTIME  TIME              1828 r   01-JAN-2000 00:00    01-JAN-2005 00:00
 normal    E
 normal    F

yes? load ttemp
 **ERROR: request exceeds memory setting: 2369 Mwords were requested.
 *** NOTE: You can use SET MEMORY/SIZE=xxx to increase memory.
 *** NOTE: The "Memory use" section of the FERRET Users Guide has further tips.

And the longer time axis:

yes? define axis/t=1-jan-2000:1-jan-2010:1/units=days  longtime
Replacing definition of axis LONGTIME

yes? show grid ttemp
    GRID (G002)
 name       axis              # pts   start                end
 XAXLEVITR LONGITUDE          360mr   20.5E                19.5E(379.5)
 YAXLEVITR LATITUDE           180 r   89.5S                89.5N
 ZAXLEVITR DEPTH (m)           20 i-  0                    5000
 LONGTIME  TIME              3654 r   01-JAN-2000 00:00    01-JAN-2010 00:00
 normal    E
 normal    F

yes? load ttemp
 **ERROR: request exceeds memory setting: 4735 Mwords were requested.
 *** NOTE: You can use SET MEMORY/SIZE=xxx to increase memory.
 *** NOTE: The "Memory use" section of the FERRET Users Guide has further tips.

[karlmsmith]

Adding @AndrewWittenberg