Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError #493

Closed
treerink opened this issue Jun 14, 2019 · 2 comments
Closed

Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError #493

treerink opened this issue Jun 14, 2019 · 2 comments

Comments

@treerink
Copy link
Collaborator

Error popping up with cmorisation of a group of not earlier cmorised NEMO variables in the optionlabel-and--messaging branch:

/nfs/home/users/reerink/cmorize/ece2cmor3/ece2cmor3/cmor_utils.py:263: RuntimeWarning: divide by zero encountered in long_scalars
  size = ncvar.size / ntimes
Traceback (most recent call last):
  File "/lustre2/projects/model_testing/reerink/anaconda2/envs/ece2cmor3/bin/ece2cmor", line 11, in <module>
    load_entry_point('ece2cmor3', 'console_scripts', 'ece2cmor')()
  File "/nfs/home/users/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmor.py", line 143, in main
    ece2cmorlib.perform_nemo_tasks(args.datadir, args.exp, refdate)
  File "/nfs/home/users/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmorlib.py", line 199, in perform_nemo_tasks
    nemo2cmor.execute(nemo_tasks)
  File "/nfs/home/users/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 113, in execute
    execute_netcdf_task(dataset, task)
  File "/nfs/home/users/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 188, in execute_netcdf_task
    time_selection=time_sel)
  File "/nfs/home/users/reerink/cmorize/ece2cmor3/ece2cmor3/cmor_utils.py", line 264, in netcdf2cmor
    chunk = int(math.floor(4.0E+9 / (8 * size)))  # Use max 4 GB of memory
ZeroDivisionError: float division by zero

Note that below tos cmorised successful, and the results were there, but cfc11 is not produced though its log below looks fine. This is the last reported variable before the crash.

2019-06-13 22:57:39 INFO:ece2cmor3.nemo2cmor: Start cmorization of tos in table 3hr
2019-06-13 22:57:39 INFO:ece2cmor3.nemo2cmor: Creating time axes for table 3hr from data in /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_3h_19900101_19901231_opa_grid_T_2D.nc...
2019-06-13 22:57:39 INFO:ece2cmor3.nemo2cmor: Creating time axis for table 3hr using file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_3h_19900101_19901231_opa_grid_T_2D.nc...
2019-06-13 22:57:40 INFO:ece2cmor3.nemo2cmor: Creating depth axes for table 3hr from data in /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_3h_19900101_19901231_opa_grid_T_2D.nc ...
2019-06-13 22:57:40 INFO:ece2cmor3.nemo2cmor: Creating depth axes for table 3hr using file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_3h_19900101_19901231_opa_grid_T_2D.nc...
2019-06-13 22:57:40 INFO:ece2cmor3.nemo2cmor: Creating extra axes for table 3hr from data in /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_3h_19900101_19901231_opa_grid_T_2D.nc ...
2019-06-13 22:57:40 INFO:ece2cmor3.nemo2cmor: Creating extra axes for table 3hr using file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_3h_19900101_19901231_opa_grid_T_2D.nc...
2019-06-13 22:57:40 INFO:ece2cmor3.nemo2cmor: Cmorizing variable tos                  in table 3hr     in file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_3h_19900101_19901231_opa_grid_T_2D.nc

2019-06-13 22:58:38 INFO:ece2cmor3.nemo2cmor: Start cmorization of cfc11,ocontempdiff,ocontemppadvect,ocontemppmdiff,ocontemprmadvect,ocontemptend,osaltdiff,osaltpadvect,osaltpmdiff,osaltrmadvect,osalttend in table Oyr
2019-06-13 22:58:38 INFO:ece2cmor3.nemo2cmor: Creating time axis for table Oyr using file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_1y_19900101_19901231_opa_grid_T_3D.nc...
2019-06-13 22:58:38 INFO:ece2cmor3.nemo2cmor: Creating depth axes for table Oyr using file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_1y_19900101_19901231_opa_grid_T_3D.nc...
2019-06-13 22:58:38 INFO:ece2cmor3.nemo2cmor: Creating extra axes for table Oyr using file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_1y_19900101_19901231_opa_grid_T_3D.nc...
2019-06-13 22:58:38 INFO:ece2cmor3.nemo2cmor: Cmorizing variable cfc11                in table Oyr     in file /lustre3/projects/CMIP6/reerink/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/t001_1y_19900101_19901231_opa_grid_T_3D.nc

Running Oyr cfc11 on another platform (KNMI-workstation), by:

ece2cmor ~/mount-point-lustre-3/ec-earth-3/branch-r6874-control-output-files/t001/output/nemo/001/ --exp t001 --nemo --meta ~/mount-point-bull/ec-earth-3/trunk/runtime/classic/ctrl/cmip6-output-control-files/CMIP/EC-EARTH-AOGCM/cmip6-experiment-CMIP-piControl/metadata-cmip6-CMIP-piControl-EC-EARTH-AOGCM-nemo-template.json --varlist ece2cmor3/resources/test-data-request/varlist-nemo-Oyr-cfc11.json --ececonf EC-EARTH-AOGCM --odir cmor-cfc11 --log

with the error:

Traceback (most recent call last):
  File "/usr/people/reerink/anaconda2/envs/ece2cmor3/bin/ece2cmor", line 11, in <module>
    load_entry_point('ece2cmor3', 'console_scripts', 'ece2cmor')()
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmor.py", line 143, in main
    ece2cmorlib.perform_nemo_tasks(args.datadir, args.exp, refdate)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmorlib.py", line 199, in perform_nemo_tasks
    nemo2cmor.execute(nemo_tasks)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 105, in execute
    create_time_axes(dataset, task_list, table)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 295, in create_time_axes
    times, time_bounds = read_times(ds, task)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 360, in read_times
    times = None if vals is None else netCDF4.num2date(vals, units=units, calendar=calendar)
  File "cftime/_cftime.pyx", line 315, in cftime._cftime.num2date
IndexError: list index out of range
@treerink treerink changed the title Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError Jun 14, 2019
@treerink treerink mentioned this issue Jun 14, 2019
Closed
@treerink treerink changed the title Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError Jun 14, 2019
@treerink treerink changed the title Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError Error with cmorising Oyr cfc11: ZeroDivisionError / IndexError Jun 14, 2019
treerink added a commit that referenced this issue Jun 14, 2019
@treerink
Add test varlist for Oyrcfc11 #493.
3b03549
@treerink
Copy link
Collaborator Author

Besdies Oyr cfc11 the same problem applies on this list of Oyr variables:

 Oyr cfc11
 Oyr ocontempdiff
 Oyr ocontemppadvect
 Oyr ocontemppmdiff
 Oyr ocontemprmadvect
 Oyr ocontemptend
 Oyr osaltdiff
 Oyr osaltpadvect
 Oyr osaltpmdiff
 Oyr osaltrmadvect
 Oyr osalttend

Note that some (new added) Oyr variables do cmorise well.

treerink added a commit that referenced this issue Jun 14, 2019
@treerink
Add a variant without the #493 problematic variables.
e1f5258
@treerink treerink mentioned this issue Jun 20, 2019
Closed
This was referenced Nov 15, 2019
Open
Closed
treerink added a commit that referenced this issue Nov 20, 2019
@treerink
Add the json data request exceptions to prevent the crash mentioned i…
8fa5bbd 
…n the 2nd post of #542. In comment the json data request exceptions for #493 & #504 are added.
treerink added a commit that referenced this issue Nov 20, 2019
@treerink
Add the Oyr cfc11-group to the exception list in drq2varlist.py #493 & …
2cf54ba 
…#542.
@treerink
Copy link
Collaborator Author

The entire group of Oyr variables mentioned in #493 (comment) only pops-up in the test-all test. Now they are added to the exception list of drq2varlist.py this issue can be closed, as these variables are further not used so far and do not longer cause a test-all test crash in this way.

treerink added a commit that referenced this issue Nov 22, 2019
@treerink
Update the varlists for excepting cfc11 & fgsf6, and the Oyr cfc11-gr…
c5512b8 
…oup #493 #504.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@treerink