Hard coded attributes

[sfinkens]

I'm trying to use CMOR3 with the obs4MIPs tables and CV:

import cmor
import numpy as np

# Init cmor
cmor.setup(netcdf_file_action=cmor.CMOR_REPLACE,
           inpath='tables')
cmor.dataset_json('user_input.json')
cmor.load_table('obs4MIPs_Amon.json')

# Define test data
time = np.array([0], dtype='f8')
lon = np.array([0, 1], dtype='f8')
lat = np.array([0, 1], dtype='f8')
time_bnds = np.array([[0, 1]], dtype='f8')
lon_bnds = np.array([[0, 1], [1, 2]], dtype='f8')
lat_bnds = np.array([[0, 1], [1, 2]], dtype='f8')
clt = np.random.rand(time.size, lat.size, lon.size).astype('f4')

# Define cmor axes
cmor_time = cmor.axis(table_entry='time',
                      units='days since 1970-01-01 00:00')
cmor_lat = cmor.axis(table_entry='latitude',
                     units='degrees_north',
                     length=lat.size,
                     coord_vals=lat,
                     cell_bounds=lat_bnds)
cmor_lon = cmor.axis(table_entry='longitude',
                     units='degrees_east',
                     length=lon.size,
                     coord_vals=lon,
                     cell_bounds=lon_bnds)

# Define cmor variable
cmor_var = cmor.variable(table_entry='clt',
                         units='1',
                         axis_ids=(cmor_time, cmor_lat, cmor_lon),
                         type='f',
                         missing_value=-999.,
                         original_name='cfc',
                         history='hello, world')

# Re-write data
cmor.write(var_id=cmor_var,
           data=clt,
           ntimes_passed=time.size,
           time_vals=time,
           time_bnds=time_bnds)
cmor.close()

This is my user_input.json

{
    "_control_vocabulary_file":     "obs4MIPs_CV.json",
    "_AXIS_ENTRY_FILE":             "obs4MIPs_coordinate.json",
    "_FORMULA_VAR_FILE":            "obs4MIPs_formula_terms.json",
    "_cmip6_option":                "CMIP6",
    "activity_id":                  "obs4MIPs",
    "calendar":                     "gregorian",

    "data_structure":               "grid",
    "dataset_version_number":       "1.1.0",
    "frequency":                    "mon",
    "grid":                         "0.5 x 0.5 degree latitude x longitude",
    "grid_label":                   "gr",
    "nominal_resolution":           "50 km",
    "product":                      "satellite",
    "realm":                        "atm",
    "region":                       "global",

    "comment":                      "",
    "contact":                      "PCMDI (pcmdi-cmip@llnl.gov) ",
    "further_info_url":             "http://www-pcmdi.llnl.gov/projects/amip/AMIP2EXPDSN/BCS/amip2bcs.php",
    "institution_id":               "RSS",
    "institution":                  "Remote Sensing Systems",
    "references":                   "",
    "source":                       "My Dataset",
    "source_id":                    "My-Dataset",
    "title":                        "My Dataset (2017)",

    "outpath":                      "./output/",
    "output_path_template":         "<activity_id><realm><frequency><variable_id><product><source_id><grid_label><version>",
    "output_file_template":         "<variable_id><table><product><source_id><grid_label>",

    "license":                      "Testing 123"
}

I generated the CV using the python script in demo1. The formula terms were copied from the CMIP6 repository as I couldn't find them here. Running the above python script throws the following errors:

C Traceback:
In function: _CV_setInstitution                                                                                                                                                                                                              
! called from: cmor_setGblAttr                                                                                                                                                                                                               
! called from: cmor_write                                                                                                                                                                                                                    
!                                                                                                                                                                                                                                            

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute institution "Remote Sensing Systems" will be replaced with 
! "Remote Sensing Systems, Santa Rosa, CA 95401, USA" as defined in your Control Vocabulary file.
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkSourceID                                                                                                                                                                                                             
! called from: cmor_setGblAttr                                                                                                                                                                                                               
! called from: cmor_write                                                                                                                                                                                                                    
!                                                                                                                                                                                                                                            

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your "source_ids" key could not be found in
! your Control Vocabulary file.(tables/obs4MIPs_CV.json)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_get_cur_dataset_attribute                                                                                                                                                                                                
! called from: _CV_checkExperiment                                                                                                                                                                                                           
! called from: cmor_setGblAttr                                                                                                                                                                                                               
! called from: cmor_write                                                                                                                                                                                                                    
!                                                                                                                                                                                                                                            

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Dataset: current dataset does not have attribute : experiment_id
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkExperiment                                                                                                                                                                                                           
! called from: cmor_setGblAttr                                                                                                                                                                                                               
! called from: cmor_write                                                                                                                                                                                                                    
!                                                                                                                                                                                                                                            

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your "experiment_ids" key could not be found in
! your Control Vocabulary file.(tables/obs4MIPs_CV.json)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_get_cur_dataset_attribute                                                                                                                                                                                                
! called from: _CV_checkParentExpID                                                                                                                                                                                                          
! called from: _CV_checkFurtherInfoURL                                                                                                                                                                                                       
! called from: cmor_setGblAttr                                                                                                                                                                                                               
! called from: cmor_write                                                                                                                                                                                                                    
!                                                                                                                                                                                                                                            

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Dataset: current dataset does not have attribute : experiment_id
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkParentExpID                                                                                                                                                                                                          
! called from: _CV_checkFurtherInfoURL                                                                                                                                                                                                       
! called from: cmor_setGblAttr                                                                                                                                                                                                               
! called from: cmor_write
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your "experiment_id" key could not be found in
! your Control Vocabulary file.(tables/obs4MIPs_CV.json)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_get_cur_dataset_attribute
! called from: _CV_checkSubExperiment
! called from: _CV_checkFurtherInfoURL
! called from: cmor_setGblAttr
! called from: cmor_write
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Dataset: current dataset does not have attribute : experiment_id
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkSubExperiment
! called from: _CV_checkFurtherInfoURL
! called from: cmor_setGblAttr
! called from: cmor_write
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your "sub_experiment_id" key could not be found in
! your Control Vocabulary file.(tables/obs4MIPs_CV.json)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_write
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Cannot continue until you fix the errors listed above: -4
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Seems like CMOR has some hard coded attribute requirements for experiment_id and sub_experiment_id , but as far as I know they are not desired in obs4MIPs. Furthermore CMOR wants to check them against the CV, but there are no such entries. Finally, CMOR wants to check source_id against the CV. Is source_id registered content as well? If so, how do I register a dataset?

[dnadeau4]

For other experiment, you need to take this out:
_cmip6_option": "CMIP6",
or set it to some other string. This triggers a bunch of verification that you do not need.

[sfinkens]

Perfect, thanks!

[durack1]

@dnadeau4 for non-cmip6 projects, should we explicitly set this entry to "_cmip6_option": "none" or something equivalent?

If yes, we'll have to add this into the demos - is there anything else missing from demo/drive_obs4MIPs.json that you can see?

[dnadeau4]

This should not be used, you can either set it to anything else than CMIP6 like none or your can just delete this input. The _ (underscore) input are use for internal purpose in CMOR and PrePARE.

[durack1]

@dnadeau4 thanks for the heads up.. @Funkensieper where did you get the _cmip6_option": "CMIP6" example from? It doesn't appear in either of the demos we have included in this repo

[sfinkens]

I think it was a leftover of the CMOR3 Python example example.

[durack1]

@Funkensieper thanks for the heads up, we'll need to clean up any of these undocumented examples.. @dnadeau4 can you take a peek to make sure the https://cmor.llnl.gov/mydoc_cmor3_python/ is up-to-date with 3.2.3?