Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

binDensity crash - check inputs #40

Open
durack1 opened this issue Apr 3, 2015 · 8 comments
Open

binDensity crash - check inputs #40

durack1 opened this issue Apr 3, 2015 · 8 comments
Assignees
@durack1
Milestone
Generalize code t...

Comments

@durack1
Copy link
Collaborator

durack1 commented Apr 3, 2015 •
edited
Loading 

FileCount:  14
outPath:    /work/durack1/Shared/150313_data_density/historicalNat
outfile:    cmip5.MRI-CGCM3.historicalNat.r1i1p1.mo.ocn.Omon.density.ver-v20120510.latestX.WARN2.xml.nc
so:         cmip5.MRI-CGCM3.historicalNat.r1i1p1.mo.ocn.Omon.so.ver-v20120510.latestX.WARN2.xml
thetao:     cmip5.MRI-CGCM3.historicalNat.r1i1p1.mo.ocn.Omon.thetao.ver-v20120510.latestX.WARN2.xml
areacello:  cmip5.MRI-CGCM3.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-v20110831.latestX.WARN2.xml
grdsize: 6756480
 ==> model: MRI-CGCM3  (grid size: 6756480 )
 ==> time interval:  0 1871
 ==> size of time chunk, number of time chunks (memory optimization) : 12 156

 --> time chunk (bounds) =  1 / 156  ( 0 11 ) MRI-CGCM3
Traceback (most recent call last):
  File ".//drive_density.py", line 262, in <module>
    densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
  File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin
    nomask      = npy.equal(vmask_3D[0],0) ; # Returns boolean
IndexError: invalid index to scalar variable.

and

FileCount:  31                                                                                                                         
outPath:    /work/durack1/Shared/150313_data_density/rcp85                                                                             
outfile:    cmip5.MRI-CGCM3.rcp85.r1i1p1.mo.ocn.Omon.density.ver-v20120510.latestX.WARN2.xml.nc                                        
so:         cmip5.MRI-CGCM3.rcp85.r1i1p1.mo.ocn.Omon.so.ver-v20120510.latestX.WARN2.xml                                                
thetao:     cmip5.MRI-CGCM3.rcp85.r1i1p1.mo.ocn.Omon.thetao.ver-v20120510.latestX.WARN2.xml                                            
areacello:  cmip5.MRI-CGCM3.sstClim4xCO2.r0i0p0.fx.ocn.fx.areacello.ver-v20110831.latestX.xml                                          
grdsize: 6756480                                                                                                                       
 ==> model: MRI-CGCM3  (grid size: 6756480 )                                                                                           
 ==> time interval:  0 1139                                                                                                            
 ==> size of time chunk, number of time chunks (memory optimization) : 12 95                                                           

 --> time chunk (bounds) =  1 / 95  ( 0 11 ) MRI-CGCM3                                                                                 
Traceback (most recent call last):                                                                                                     
  File ".//drive_density.py", line 264, in <module>                                                                                    
    densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')                                                     
  File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin                                                     
    nomask      = npy.equal(vmask_3D[0],0) ; # Returns boolean                                                                         
IndexError: invalid index to scalar variable.                                                                                          
[durack1@crunchy Density_bining]$ cl
@durack1 durack1 self-assigned this Apr 3, 2015
@durack1
Copy link
Collaborator Author

durack1 commented Apr 13, 2015

Same issue, historical experiment:

 Wrote file:  /work/durack1/Shared/150313_data_density/historical/cmip5.MPI-ESM-P.historical.r1i1p1.an.ocn.Omon.density.ver-1.nc     
FileCount:  45                                                                                                                       
outPath:    /work/durack1/Shared/150313_data_density/historical                                                                      
outfile:    cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.density.ver-v20120510.latestX.WARN2.xml.nc                                 
so:         cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.so.ver-v20120510.latestX.WARN2.xml                                         
thetao:     cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.thetao.ver-v20120510.latestX.WARN2.xml                                     
areacello:  cmip5.MRI-CGCM3.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-v20110831.latestX.WARN2.xml                                         
grdsize: 6756480                                                                                                                     
 ==> model: MRI-CGCM3  (grid size: 6756480 )                                                                                         
 ==> time interval:  0 1871                                                                                                          
 ==> size of time chunk, number of time chunks (memory optimization) : 12 156                                                        

 --> time chunk (bounds) =  1 / 156  ( 0 11 ) MRI-CGCM3                                                                              
Traceback (most recent call last):                                                                                                   
  File ".//drive_density.py", line 262, in <module>                                                                                  
    densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')                                                   
  File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin                                                   
    nomask      = npy.equal(vmask_3D[0],0) ; # Returns boolean                                                                       
IndexError: invalid index to scalar variable.

@eguil
Copy link
Owner

eguil commented Apr 13, 2015

ok I will look into this.

On 13/4/15 19:38, Paul J. Durack wrote:

Same issue, historical experiment:

| Wrote file: /work/durack1/Shared/150313_data_density/historical/cmip5.MPI-ESM-P.historical.r1i1p1.an.ocn.Omon.density.ver-1.nc
FileCount: 45
outPath: /work/durack1/Shared/150313_data_density/historical
outfile: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.density.ver-v20120510.latestX.WARN2.xml.nc
so: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.so.ver-v20120510.latestX.WARN2.xml
thetao: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.thetao.ver-v20120510.latestX.WARN2.xml
areacello: cmip5.MRI-CGCM3.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-v20110831.latestX.WARN2.xml
grdsize: 6756480
==> model: MRI-CGCM3 (grid size: 6756480 )
==> time interval: 0 1871
==> size of time chunk, number of time chunks (memory optimization) : 12 156

--> time chunk (bounds) = 1 / 156 ( 0 11 ) MRI-CGCM3
Traceback (most recent call last):
File ".//drive_density.py", line 262, in
densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin
nomask = npy.equal(vmask_3D[0],0) ; # Returns boolean
IndexError: invalid index to scalar variable.
|

�
Reply to this email directly or view it on GitHub
#40 (comment).

Eric Guilyardi
IPSL/LOCEAN - Dir. Rech. CNRS
Tour 45, 4eme, piece 406
UPMC, case 100
4 place Jussieu, F-75252 Paris
Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi
NCAS Climate
Meteorology Department
University of Reading
Reading RG6 6BB - UK
Tel: +44 (0)118 378 8315

             http://ncas-climate.nerc.ac.uk/~ericg

@durack1
Copy link
Collaborator Author

durack1 commented Apr 13, 2015

@eguil these problems are most likely the inputs - the files include the .WARN2. identifier - so just leave this up to me.. I'm just restarting for the moment, will have to go and revisit once I get to the end of the available files.. We're still running out of memory on MIROC4h and MPI.. But again will just keep pushing this until I get to the end before digging deeper..

@eguil
Copy link
Owner

eguil commented Apr 13, 2015

@durack1 ok fine

On 13/4/15 23:14, Paul J. Durack wrote:

@eguil https://github.com/eguil these problems are most likely the
inputs - the files include the |.WARN2.| identifier - so just leave
this up to me.. I'm just restarting for the moment, will have to go
and revisit once I get to the end of the available files.. We're still
running out of memory on MIROC4h and MPI.. But again will just keep
pushing this until I get to the end before digging deeper..

�
Reply to this email directly or view it on GitHub
#40 (comment).

Eric Guilyardi
IPSL/LOCEAN - Dir. Rech. CNRS
Tour 45, 4eme, piece 406
UPMC, case 100
4 place Jussieu, F-75252 Paris
Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi
NCAS Climate
Meteorology Department
University of Reading
Reading RG6 6BB - UK
Tel: +44 (0)118 378 8315

             http://ncas-climate.nerc.ac.uk/~ericg

@durack1
Copy link
Collaborator Author

durack1 commented Apr 20, 2015

And another: historicalNat
cmip5.CESM1-CAM5.historicalNat.r1i1p1.an.ocn.Omon.density.ver-v20130909.nc

 --> time chunk (bounds) =  88 / 156  ( 1044 1055 ) CESM1-CAM5
*THETAO mean: +281.82; min: +270.23; max: +306.29
*THETAO mean: +08.67; min: -02.92; max: +33.14
  Total volume in z coordinates source grid (ref = 1.33 e+18) :  1.32186397697e+18
  Mean Temp./Salinity in z coordinates source grid            :  3.49228724533 34.7171399393
 Bining CPU analysis tc/tcdel =  87 12
    average cpu1  =  0.542500000001
    average cpu2  =  2.275
    average cpu3  =  0.321666666667
    average cpu4  =  27.07
    average cpu40 =  0.0908333333349
    average cpu5  =  0.598333333332
    CPU read T/S  =  7.07
    CPU comp. rho =  13.21
 Integral persistent values: 1.24750987849e+18 1.19516e+18 3.01982395738e+17
   % 95.8034683786 25.2671582723
  area global/atl/pac/ind  3.44248e+14 9.7772e+13 1.73523e+14 7.29536e+13
   T , S glob  3.44037747224 33.0646674819
   T , S atl   3.86833315611 31.8864366555
   T , S pac   3.09515285816 33.3947889029
   T , S ind   3.68796375976 33.8585274262
 Persistence CPU analysis t/nyrtc =  0 1
    cpu1 =  21.74
    cpu2 =  0.0
    cpu3 =  0.0
    cpu4 =  0.0
    cpu5 =  0.0
    cpu6 =  0.0
    cpu7 =  0.0
   CPU of chunk inits         = 21.91
   CPU of density bining      = 370.78
   CPU of masking and var def = 8.13
   CPU of annual mean compute = 4.83
   CPU of interpolation       = 5.96
   CPU of zonal mean          = 2.65
   CPU of persistence compute = 21.74
   CPU of chunk               = 436.0
   Max memory use 17.77614 GB
 --> time chunk (bounds) =  89 / 156  ( 1056 1067 ) CESM1-CAM5
*THETAO mean: +281.82; min: +270.04; max: +305.50
*THETAO mean: +08.67; min: -03.11; max: +32.35
  Total volume in z coordinates source grid (ref = 1.33 e+18) :  1.32186397697e+18
  Mean Temp./Salinity in z coordinates source grid            :  3.49275783464 34.7171368331
 Bining CPU analysis tc/tcdel =  88 12
    average cpu1  =  0.176666666667
    average cpu2  =  2.17083333333
    average cpu3  =  0.286666666666
    average cpu4  =  20.3491666667
    average cpu40 =  0.0716666666667
    average cpu5  =  0.449166666665
    CPU read T/S  =  3.81
    CPU comp. rho =  14.36
Traceback (most recent call last):
  File ".//drive_density.py", line 262, in <module>
    densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
  File "/export/durack1/git/Density_bining/binDensity.py", line 969, in densityBin
    maskp = npy.ma.reshape(maskp, (N_s+1, latN*lonN))
  File "/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/numpy/ma/core.py", line 6508, in reshape
    return a.reshape(new_shape, order=order)
ValueError: total size of new array must be unchanged

@durack1
Copy link
Collaborator Author

durack1 commented Apr 22, 2015

These two issues are likely due to non-z vertical coordinates..

And another (this issue is also the case with r2i1p1 through r5i1p1):

FileCount:  41
outPath:    /work/durack1/Shared/150313_data_density/historical
outfile:    cmip5.MIROC5.historical.r1i1p1.mo.ocn.Omon.density.ver-1.nc
so:         cmip5.MIROC5.historical.r1i1p1.mo.ocn.Omon.so.ver-1.latestX.xml
thetao:     cmip5.MIROC5.historical.r1i1p1.mo.ocn.Omon.thetao.ver-1.latestX.xml
areacello:  cmip5.MIROC5.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-1.latestX.xml
grdsize: 2867200
 ==> model: MIROC5  (grid size: 2867200 )
 ==> time interval:  0 1955
 ==> size of time chunk, number of time chunks (memory optimization) : 12 163
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1241: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if (srcMaskValues != None):
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1248: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if (dstMaskValues != None):

 --> time chunk (bounds) =  1 / 163  ( 0 11 ) MIROC5
Traceback (most recent call last):
  File ".//drive_density.py", line 262, in <module>
    densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
  File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin
    nomask      = npy.equal(vmask_3D[0],0) ; # Returns boolean
IndexError: invalid index to scalar variable.

And another:

FileCount:  46
outPath:    /work/durack1/Shared/150313_data_density/historical
outfile:    cmip5.MRI-ESM1.historical.r1i1p1.mo.ocn.Omon.density.ver-v20130307.latestX.WARN2.xml.nc
so:         cmip5.MRI-ESM1.historical.r1i1p1.mo.ocn.Omon.so.ver-v20130307.latestX.WARN2.xml
thetao:     cmip5.MRI-ESM1.historical.r1i1p1.mo.ocn.Omon.thetao.ver-v20130307.latestX.WARN2.xml
areacello:  cmip5.MRI-ESM1.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-v20140210.latestX.xml
grdsize: 6756480
 ==> model: MRI-ESM1  (grid size: 6756480 )
 ==> time interval:  0 1859
 ==> size of time chunk, number of time chunks (memory optimization) : 12 155
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1241: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if (srcMaskValues != None):
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1248: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if (dstMaskValues != None):

 --> time chunk (bounds) =  1 / 155  ( 0 11 ) MRI-ESM1
Traceback (most recent call last):
  File ".//drive_density.py", line 262, in <module>
    densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
  File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin
    nomask      = npy.equal(vmask_3D[0],0) ; # Returns boolean
IndexError: invalid index to scalar variable.

@durack1
Copy link
Collaborator Author

durack1 commented Jul 20, 2015

And another:

FileCount:  152
outPath:    /work/durack1/Shared/150313_data_density/historical
outfile:    cmip5.MIROC4h.historical.r2i1p1.mo.ocn.Omon.density.ver-1.nc
so:         cmip5.MIROC4h.historical.r2i1p1.mo.ocn.Omon.so.ver-1.latestX.xml
thetao:     cmip5.MIROC4h.historical.r2i1p1.mo.ocn.Omon.thetao.ver-1.latestX.xml
areacello:  cmip5.MIROC4h.rcp45.r0i0p0.fx.ocn.fx.areacello.ver-1.latestX.xml
grdsize: 56033280
 ==> model: MIROC4h  (grid size: 56033280 )
 ==> time interval:  0 671
 ==> size of time chunk, number of time chunks (memory optimization) : 12 56

 --> time chunk (bounds) =  1 / 56  ( 0 11 ) MIROC4h
trigger mask fix - EC-EARTH/MIROC4h
350524236
*THETAO mean: +281.47; min: +271.11; max: +309.97
*THETAO mean: +08.33; min: -02.04; max: +36.82
  Total volume in z coordinates source grid (ref = 1.33 e+18) :  1.36475536317e+18
  Mean Temp./Salinity in z coordinates source grid            :  4.03157081096 34.7199550623
 Bining CPU analysis tc/tcdel =  0 12
    average cpu1  =  1.42166666667
    average cpu2  =  21.6533333333
    average cpu3  =  2.38583333333
    average cpu4  =  236.875
    average cpu40 =  1.09749999999
    average cpu5  =  4.55416666667
    CPU read T/S  =  283.05
    CPU comp. rho =  248.47
0.0 0.0 0.0
0.0 0.0 0.0
  Total volume in rho coordinates source grid (ref = 1.33 e+18) :  1.31614338386e+18
  Mean Temp./Salinity in rho coordinates source grid            :  4.06755384743 34.8811525226
Traceback (most recent call last):
  File ".//drive_density.py", line 262, in <module>
    densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
  File "/export/durack1/git/Density_bining/binDensity.py", line 961, in densityBin
    idxvm = 1-mv.masked_values(thickBin[inim:finm,:,:,:], valmask).mask
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/cdms2/MV2.py", line 867, in masked_values
    maresult = numpy.ma.masked_values(_makeMaskedArg(data), value, rtol=rtol, atol=atol, copy=copy)
  File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/numpy/ma/core.py", line 2193, in masked_values
    condition = umath.less_equal(mabs(xnew - value), atol + rtol * mabs(value))
MemoryError

@durack1
Copy link
Collaborator Author

durack1 commented Jul 26, 2017

Revisit as specific test cases in #33 - test with new code and new ESMPy

@durack1 durack1 added this to the Generalize code toward v2.0 milestone Jul 26, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@durack1 durack1

Labels
None yet
Projects
None yet
Milestone
Generalize code toward v2.0
Development

No branches or pull requests
2 participants
@durack1 @eguil