Convert velocity input to optional (obs have no velocity input) #74

durack1 · 2019-02-15T06:56:22Z

EN4 and other obs datasets have no corresponding velocity time series. Consequently current code won’t process obs.

eguil · 2019-02-18T22:09:39Z

Option added and input arguments slightly revisited:

densityBin(fileT,fileS,fileFx,fileV='none',outFile='out.nc',debug=True,timeint='all',\
mthout=False,gridfT='none',gridfS='none',gridfV='none'):

if fileV='none' the MSF calculation is ignored and V read is not required.

eguil · 2019-02-18T23:15:48Z

Tested and ok

durack1 · 2019-02-28T01:25:21Z

@eguil I'm still hitting problems with this:

(cdat80py2) duro@ocean:[Density_bining]:[issue71_durack1_UpdateObsUpdateESMPCalls*]:[11638]> python drive_obs.py
Processing:  0
areacello:  ~Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.so.ver-1.latestX.xml
so:         ~Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.thetao.ver-1.latestX.xml
thetao:     ~Shared/190213_data_density/obs.JAMSTEC.historical.r0i0p0.mo.ocn.Omon.so.ver-1.latestX.xml
Warning: ID reference not found: bounds=bounds_latitude
Warning: ID reference not found: bounds=bounds_longitude
Warning: ID reference not found: bounds=bounds_pressure
Warning: ID reference not found: bounds=bounds_time
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Traceback (most recent call last):
  File "drive_obs.py", line 87, in <module>
    densityBin(thetao,so,areacello,'none',outfileDensity,debug=True,timeint='all')
  File "~git/Density_bining/binDensity.py", line 358, in densityBin
    thetao_h    = ft('thetao', time = slice(1,10)) ; # remove handle for non cmor files
  File "~anaconda2/envs/cdat80py2/lib/python2.7/site-packages/cdms2/cudsinterface.py", line 33, in __call__
    raise CDMSError("No such variable, " + id)
cdms2.error.CDMSError: No such variable, thetao

eguil · 2019-03-05T22:15:15Z

@durack1 can you put the files somewhere on crunchy ? It is probably due to non CMOR format. Is the temperature variable name thetao ?

durack1 · 2019-03-05T22:52:09Z

@eguil it still seems to be an issue:

(cdat80py2) durack1@oceanonly:[Density_bining]:[issue71_durack1_UpdateObsUpdateESMPCalls*]:[11506]> python drive_obs.py
Processing:  0
areacello:  /work/durack1/Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.areacello.ver-1.latestX.nc
so:         /work/durack1/Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.so.ver-1.latestX.xml
thetao:     /work/durack1/Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.thetao.ver-1.latestX.xml
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Traceback (most recent call last):
  File "drive_obs.py", line 89, in <module>
    densityBin(thetao,so,areacello,'none',outfileDensity,debug=True,timeint='all')
  File "/export/durack1/git/Density_bining/binDensity.py", line 364, in densityBin
    vo_h        = fv('vo'    , time = slice(1,10)) ; #
  File "/export/durack1/anaconda2/envs/cdat80py2/lib/python2.7/site-packages/cdms2/cudsinterface.py", line 33, in __call__
    raise CDMSError("No such variable, " + id)
cdms2.error.CDMSError: No such variable, vo

I'll email the paths to data

eguil · 2019-03-05T23:26:50Z

is the call you make to density bin correct (arguments have been re-organised) ?

durack1 · 2019-03-05T23:31:02Z

@eguil looks right to me:

densityBin(thetao,so,areacello,'none',outfileDensity,debug=True,timeint='all')

https://github.com/eguil/Density_bining/blob/master/binDensity.py#L244-L266

def densityBin(fileT,fileS,fileFx,fileV='none',outFile='out.nc',debug=True,timeint='all',mthout=False,gridfT='none',gridfS='none',gridfV='none'):
    '''
    The densityBin() function takes file and variable arguments and creates
    density persistence fields which are written to a specified outfile
    Author:    Eric Guilyardi : Eric.Guilyardi@locean-ipsl.upmc.fr
    Co-author: Paul J. Durack : pauldurack@llnl.gov : @durack1.
    Created on Sun Sep 14 21:13:30 2014
    Inputs:
    ------
    - fileT(time,lev,lat,lon)   - 4D potential temperature array
    - fileS(time,lev,lat,lon)   - 4D salinity array
    - fileFx(lat,lon)           - 2D array containing the cell area values
    -> options:
    - fileV(time,lev,lat,lon)   - 4D meridional velocity array
    - outFile(str)              - output file with full path specified.
    - debug <optional>          - boolean value
    - timeint <optional>        - specify temporal step for binning <init_idx>,<ncount>
    - mthout <optional>         - write annual data (False) or all monthly data (True)
    - gridfT <optional>         - file to get T grid info from
    - gridfS <optional>         - file to get S grid info from
    - gridfV <optional>         - file to get V grid info from
...

eguil · 2019-03-07T11:17:16Z

ok I managed to run my latest version of binDensity (i.e. without your last changes) on the first year of EN4 and it is running. Here is my test script:

from binDensity import densityBin

obsThetao = '/data/ericglod/Density_binning/190213_data_density/180723_EN4.2.1.g10_monthly_TAndS_190001-190912_5-5350m.nc'
obsSo = '/data/ericglod/Density_binning/190213_data_density/180723_EN4.2.1.g10_monthly_TAndS_190001-190912_5-5350m.nc'
obsAreacello = '/data/ericglod/Density_binning/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.areacello.ver-1.latestX.nc'

outfileDensity = '/home/ericglod/Density_bining/test/EN4.mon.ocean.Omon.density.nc'

densityBin(obsThetao,obsSo,obsAreacello,'none',outFile=outfileDensity,timeint='1,12')

Not sure what is going on on your side. Maybe we should do a call with screen sharing.

durack1 · 2019-03-07T17:00:55Z

@eguil what the the version of CDAT that you're running this against? A conda env?

eguil · 2019-03-07T17:47:47Z

an old PMP installation version

durack1 · 2019-03-07T17:56:14Z

Got any more details? If it's conda (probably not), you can peruse through /work/eguilyardi/anaconda2/envs/ - if it's CICLAD (looks like it), I'm not sure..

eguil · 2019-03-07T18:09:22Z

I am running on ciclad and it is not a conda install.

durack1 · 2019-03-07T20:49:31Z

Probably time to make the change #69, download anaconda

conda create -n cdat81py2 -c cdat/label/v81 -c conda-forge cdat "python<3"

ysilvy · 2019-03-07T21:31:27Z

Eric I don’t know if this can help but we can actually easily create conda environments on Ciclad, I’ve been testing it with simple stuff and installing new packages which seems to work, should we test that option ? That way we should be able to install the new version of cdat

…

Le 7 mars 2019 à 21:49, Paul J. Durack ***@***.***> a écrit : Probably time to make the change, download anaconda <https://www.anaconda.com/distribution/#download-section> conda create -n cdat81py2 -c cdat/label/v81 -c conda-forge cdat "python<3" — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#74 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWkPCkK_2hBNuoejAXVLNwkwE5rMCvImks5vUXtcgaJpZM4a9Dcp>.

durack1 · 2019-03-07T23:42:20Z

@ysilvy it's a good move to update yourselves and the way you work. Creating conda environments means that you can have versions side by side, which is a real advantage, particularly as new/beta releases become available where you can validate older libraries/code works

eguil · 2019-03-15T15:42:48Z

Ok I managed to sync again binDensity on Crunchy. I will try there.

durack1 · 2019-03-15T20:38:51Z

@eguil let me know

eguil · 2019-03-16T22:03:03Z

Ok let's try that (I will transfer all the EN4 data). I can now sync git on crunchy but the python is now v3 and binDensity is v2. Where is the cdat that would work for python2 ?

eguil · 2019-03-16T23:12:08Z

the program is running on ciclad (old set up) but I have issues with the negative masking values in En4. Is it easy to change this in the files or should I work around in the code ?

durack1 · 2019-03-17T00:00:55Z

@eguil make sure you included the "python<3" from the #74 (comment), otherwise it will default to py3

eguil · 2019-03-17T13:29:41Z

I know this is confusing but my previous comment was for my tests on crunchy where I did not install CDAT.

durack1 · 2019-03-18T15:30:34Z

@eguil I'm having trouble following the thread of this discussion above, facetime/skype?

eguil · 2019-04-30T09:58:41Z

closing this

eguil mentioned this issue Feb 18, 2019

Bug in integrated zonal mean #71

Closed

eguil closed this as completed Feb 18, 2019

durack1 reopened this Feb 28, 2019

durack1 mentioned this issue Mar 5, 2019

Issue71 durack1 update obs update esmp calls #76

Merged

eguil closed this as completed Apr 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert velocity input to optional (obs have no velocity input) #74

Convert velocity input to optional (obs have no velocity input) #74

durack1 commented Feb 15, 2019

eguil commented Feb 18, 2019 •

edited

Loading

eguil commented Feb 18, 2019

durack1 commented Feb 28, 2019

eguil commented Mar 5, 2019

durack1 commented Mar 5, 2019

eguil commented Mar 5, 2019

durack1 commented Mar 5, 2019

eguil commented Mar 7, 2019

durack1 commented Mar 7, 2019

eguil commented Mar 7, 2019

durack1 commented Mar 7, 2019

eguil commented Mar 7, 2019

durack1 commented Mar 7, 2019 •

edited

Loading

ysilvy commented Mar 7, 2019 via email

durack1 commented Mar 7, 2019

eguil commented Mar 15, 2019

durack1 commented Mar 15, 2019

eguil commented Mar 16, 2019

eguil commented Mar 16, 2019

durack1 commented Mar 17, 2019

eguil commented Mar 17, 2019

durack1 commented Mar 18, 2019

eguil commented Apr 30, 2019

Convert velocity input to optional (obs have no velocity input) #74

Convert velocity input to optional (obs have no velocity input) #74

Comments

durack1 commented Feb 15, 2019

eguil commented Feb 18, 2019 • edited Loading

eguil commented Feb 18, 2019

durack1 commented Feb 28, 2019

eguil commented Mar 5, 2019

durack1 commented Mar 5, 2019

eguil commented Mar 5, 2019

durack1 commented Mar 5, 2019

eguil commented Mar 7, 2019

durack1 commented Mar 7, 2019

eguil commented Mar 7, 2019

durack1 commented Mar 7, 2019

eguil commented Mar 7, 2019

durack1 commented Mar 7, 2019 • edited Loading

ysilvy commented Mar 7, 2019 via email

durack1 commented Mar 7, 2019

eguil commented Mar 15, 2019

durack1 commented Mar 15, 2019

eguil commented Mar 16, 2019

eguil commented Mar 16, 2019

durack1 commented Mar 17, 2019

eguil commented Mar 17, 2019

durack1 commented Mar 18, 2019

eguil commented Apr 30, 2019

eguil commented Feb 18, 2019 •

edited

Loading

durack1 commented Mar 7, 2019 •

edited

Loading