Errors and error messages in CMOR via Python

[ehogan]

I have a few issues / questions about errors and error messages in CMOR via Python; if they should be split into separate issues, please let me know.

I am currently trying to catch messages from CMOR so I can log them via the Python logger. During my experiments I discovered there are cases where errors terminate the Python interpreter. For example:

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor
No module named cdms2

In [2]: cmor.load_table('non_existent_table_name')
C Traceback:
In function: cmor_is_setup
called from: cmor_load_table


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                                    !
! Error: You need to run cmor_setup before calling any cmor_function !
!                                                                    !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

% 

I wondered if this was the case for all errors from CMOR, but I think this first example is an exception (no pun intended) ;) :

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor
No module named cdms2

In [2]: cmor.setup()

In [3]: cmor.load_table('non_existent_table_name')

C Traceback:
In function: cmor_load_table


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                      !
! Error: Could not find table: non_existent_table_name !
!                                                      !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-3-f6298efc3e7d> in <module>()
----> 1 cmor.load_table('non_existent_table_name')

/path/lib/python/cmor/pywrapper.pyc in load_table(table)
    748 ##     if not os.path.exists(table):
    749 ##         raise Exception, "Error, the table you specified (%s) does not exists" % table
--> 750     return _cmor.load_table(table)
    751 
    752 def dataset(experiment_id,institution,source,calendar,outpath='.',realization=1,contact="",history="",comment="",references="",leap_year=0,leap_month=0,month_lengths=None,model_id="",forcing="",initialization_method=None,physics_version=None,institute_id="",parent_experiment_id="",branch_time=None,parent_experiment_rip=""):

SystemError: error return without exception set

In [4]: 

Google tells me that the SystemError: error return without exception set exception can happen if code in a C extension module returns NULL (to indicate an error has occurred) but no exception was set. This means I can wrap the CMOR call in a try-except block, but the exception message is not the same as the error message printed by CMOR:

In [4]: try:
   ...:     cmor.load_table('non_existent_table_name')
   ...: except Exception, err:
   ...:     raise err
   ...:     

C Traceback:
In function: cmor_load_table


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                      !
! Error: Could not find table: non_existent_table_name !
!                                                      !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-5-a4c6935cf1f8> in <module>()
      2     cmor.load_table('non_existent_table_name')
      3 except Exception, err:
----> 4     raise err
      5 

SystemError: error return without exception set

In [5]: 

Additionally, it appears that the error message from the CMOR call isn't sent to stdout or stderr (my attempts to capture the messages have failed), so even if I use a try-except block I still get an error message printed to screen.

So, to summarise:

1. Would it be possible to fix the following issues:

• the cases where CMOR errors terminate the Python interpreter
• the cases where CMOR errors do not return a valid exception

2. Is there a way to return the error messages from CMOR in such a way that enables them to be captured by external programs?

[ehogan]

I just noticed that the points in 1) are already mentioned in issue #19, so I apologise for the duplication.

[dnadeau4]

ehogan,

You need to tell cmor where your tables are so that they can be loaded. The tables such as CMIP5_Amon, CMIP5_Omon can be found here: https://github.com/PCMDI/cmip5-cmor-tables/tree/master/Tables

Here is an example on cmor setup

        # -----------------------------------
        # Take care of cmor initialization.
        # -----------------------------------
        cmor.setup(inpath="/home/ehogan",
                   netcdf_file_action = cmor.CMOR_REPLACE)

You can find many examples in the github repository. Here is a good one that might help you setup cmor:
https://github.com/PCMDI/cmor/blob/master/Test/test_python_1D_var.py

A exhaustive documentation of each function and parameters can be found here:

https://github.com/PCMDI/cmor/blob/master/Doc/cmor_users_guide.doc

Hope this gets you going!
Denis

[doutriaux1]

@dnadeau4 we should use:
https://docs.python.org/2/c-api/exceptions.html#c.PyErr_SetString

[durack1]

@ehogan @dnadeau4 is this a dupe of #65 and if so want to close?

[ehogan]

Closing (duplicate of #65)