Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error handling with CVs #374

Closed
ehogan opened this issue Jul 24, 2018 · 5 comments
Closed

Error handling with CVs #374

ehogan opened this issue Jul 24, 2018 · 5 comments
Assignees
Projects

Comments

@ehogan
Copy link
Contributor

@ehogan ehogan commented Jul 24, 2018

Running CMOR 3.3.3 with version 01.00.27b1 of the MIP tables for a single variable works, even though the cmor log contains the following error:

C Traceback:
! In function: _CV_ValidateAttribute
! called from: _CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The attribute "data_specs_version" could not be validated. 
! The current input value is "01.00.27b1" which is not valid 
! Valid values must match the regular expression:
! 	["^[[:digit:]]\{2,2\}\.[[:digit:]]\{2,2\}\.[[:digit:]]\{2,2\}$" ] 
! 
! Check your Control Vocabulary file "Tables/01.00.27b1/CMIP6_CV.json".
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please fix required attributes mentioned in
! the warnings/error above and rerun. (aborting!)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!

This error only causes CMOR to fail when attempting to produce more than one variable:

C Traceback:
! In function: _CV_ValidateAttribute
! called from: _CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The attribute "data_specs_version" could not be validated. 
! The current input value is "01.00.27b1" which is not valid 
! Valid values must match the regular expression:
! 	["^[[:digit:]]\{2,2\}\.[[:digit:]]\{2,2\}\.[[:digit:]]\{2,2\}$" ] 
! 
! Check your Control Vocabulary file "Tables/01.00.27b1/CMIP6_CV.json".
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please fix required attributes mentioned in
! the warnings/error above and rerun. (aborting!)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_ValidateAttribute
! called from: _CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The attribute "data_specs_version" could not be validated. 
! The current input value is "01.00.27b1" which is not valid 
! Valid values must match the regular expression:
! 	["^[[:digit:]]\{2,2\}\.[[:digit:]]\{2,2\}\.[[:digit:]]\{2,2\}$" ] 
! 
! Check your Control Vocabulary file "Tables/01.00.27b1/CMIP6_CV.json".
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please fix required attributes mentioned in
! the warnings/error above and rerun. (aborting!)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Cannot continue until you fix the errors listed above: -1
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Note the additional message Error: Cannot continue until you fix the errors listed above: -1 in the more than one variable case. The first variable is produced sucessfully, but not the second.

Here is a test to reproduce this problem:

#!/usr/bin/env python2.7
# If this example is not executed from the directory containing the
# CMOR code, please first complete the following steps:
#
#   1. In any directory, create 'Tables/', 'Test/' and 'CMIP6/'
#      directories.
#
#   2. Download CMIP6_6hrLev.json, CMIP6_CV.json,
#      CMIP6_coordinate.json and CMIP6_formula_terms.json from
#      https://github.com/PCMDI/cmip6-cmor-tables/tree/master/Tables to
#      the 'Tables/' directory.
#
#   3. Download common_user_input.json from
#      https://github.com/PCMDI/cmor/blob/master/Test/ to the 'Test/'
#      directory.
import numpy

import cmor


def main():
    inpath = 'Tables'  # 01.00.27b1
    cmor.setup(inpath=inpath, netcdf_file_action=cmor.CMOR_REPLACE,
               logfile='cmor.log')
    error_flag = cmor.dataset_json('Test/common_user_input.json')
    table_id = cmor.load_table('CMIP6_6hrLev.json')
    time = cmor.axis(table_entry='time1', units='days since 2000-01-01',
                     coord_vals=numpy.array(range(1)),
                     cell_bounds=numpy.array(range(2)))
    latitude = cmor.axis(table_entry='latitude', units='degrees_north',
                         coord_vals=numpy.array(range(5)),
                         cell_bounds=numpy.array(range(6)))
    longitude = cmor.axis(table_entry='longitude', units='degrees_east',
                          coord_vals=numpy.array(range(5)),
                          cell_bounds=numpy.array(range(6)))
    plev3 = cmor.axis(table_entry='plev3', units='Pa',
                      coord_vals=numpy.array([85000., 50000., 25000.]))
    axis_ids = [longitude, latitude, plev3, time]
    ua_var_id = cmor.variable(table_entry='ua', axis_ids=axis_ids,
                              units='m s-1')
    ta_var_id = cmor.variable(table_entry='ta', axis_ids=axis_ids,
                              units='K')
    data = numpy.random.random(75)
    reshaped_data = data.reshape((5, 5, 3, 1))
    
    # This works:
    cmor.write(ua_var_id, reshaped_data)

    # This doesn't:
    #cmor.write(ta_var_id, reshaped_data)
    #cmor.write(ua_var_id, reshaped_data)

    cmor.close()

if __name__ == '__main__':
    main()

I expect this demonstrates the issue I was attempting to explain in #245.

Note I wouldn't intend to use the beta version of the MIP tables for production; I am just using them in this example, since it demonstrates the problem well.

@doutriaux1

This comment has been minimized.

Copy link
Contributor

@doutriaux1 doutriaux1 commented Jul 24, 2018

Hi @ehogan thanks for reporting this. I will fix this promptly! I really appreciate the detailed error report and test you provided! It will save us a lot of time.

@ehogan

This comment has been minimized.

Copy link
Contributor Author

@ehogan ehogan commented Jul 24, 2018

No problem! :)

@doutriaux1 doutriaux1 added this to Needed in 3.4 via automation Jul 30, 2018
@doutriaux1 doutriaux1 self-assigned this Jul 30, 2018
@doutriaux1

This comment has been minimized.

Copy link
Contributor

@doutriaux1 doutriaux1 commented Jul 30, 2018

@ehogan thanks for catching this again. I looked into this.
good news one is that the error is not really important it is due to that 27b1 is not valid should be 27 really, but we're waiting i=on the final version.
good news two is that in my "3.4" future branch (probably also in nightly tonight). This is actually intercepted in python via:

import signal
signal.signal(signal.SIGINT, self.signal_handler)
try:
   # your code here
except (KeyboardInterrupt, BaseException):
    print "Intercepted!!!!!!"

This change is necessary for fortran to exit nicely and will be documented and advertised before the next release.

As far as this bug go, @taylor13 I will add a test for it, but should this really be an error or should it be a warning?

doutriaux1 added a commit that referenced this issue Jul 30, 2018
fix #374 catch unexpected errors and check bad specs
3.4 automation moved this from Needed to Done Jul 30, 2018
@ehogan

This comment has been minimized.

Copy link
Contributor Author

@ehogan ehogan commented Jul 31, 2018

Hi @doutriaux1, I just tried testing master but it's not doing as I would expect. My apologies; I realise I wasn't very clear in my first comment.

The case that I've marked This works in the example shouldn't actually work; this case should fail and CMOR should exit, since an error is raised.

I've submitted a pull request to fix the problem: #381 :)

@doutriaux1

This comment has been minimized.

Copy link
Contributor

@doutriaux1 doutriaux1 commented Jul 31, 2018

@ehogan awesome! thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
3.4
  
Done
2 participants
You can’t perform that action at this time.