CMIP collection input file structure

[andersy005]

@matt-long, as you pointed out in #35 (comment), it is going to be a challenge to decide what goes in to a CMIP collection input file equivalent to what we have for CESM. I've started looking into this but I just realized that it's better to bring it to your attention so that you can help me with an overall structure that is robust.

So far, I have the following and I am wondering if this is how we might want to do it? What would like to see in this collection input file?

name: cmip5_test_collection
collection_type: cmip
overwriting_existing: true
include_cache_dir : true
data_sources:
  BCC:
    locations:
      - name: GLADE
        loc_type: posix
        direct_access: True 
        urlpath:
          - /glade/collections/cmip/cmip5/output1/BCC
          - /glade/collections/cmip/cmip5/output2/BCC

  BNU:
    locations:
      - name: GLADE
        loc_type: posix
        direct_access: True 
        urlpath:
          - /glade/collections/cmip/cmip5/output1/BNU
          - /glade/collections/cmip/cmip5/output2/BNU

  CCCma:
    locations:
      - name: GLADE
        loc_type: posix
        direct_access: True 
        urlpath:
          - /glade/collections/cmip/cmip5/output1/CCCma
          - /glade/collections/cmip/cmip5/output2/CCCma

Your feedback/suggestions are appreciated.

Originally posted by @andersy005 in #38 (comment)

[matt-long]

I think we have to consider that this file might play a fundamentally different role than in the CESM context. In CMIP, we have been operating in pure "discovery mode", leveraging the directory structure to infer the descriptors for each file (not so for CESM). In your example, the highest level key under data_sources is model. That already deviates from the CESM format, where the highest level key is experiment.

I think we should consider sticking with the CESM rubric, and doing something like

data_sources:
  PIcontrol: 
    locations:
      - name: GLADE
        loc_type: posix
        direct_access: True 
        urlpath:
         - /glade/collections/cmip/cmip5
    case_members:
       - model: BCC
       - model: BNU

  esmHistorical:
    locations:
      - name: GLADE
        loc_type: posix
        direct_access: True 
        urlpath:
         - /glade/collections/cmip/cmip5
    case_members:
       - model: BCC
       - model: BNU

We might consider implementing an optional locations at a level parallel with data_sources. To do this, I suggest introducing a df_files_common that is populated by the high-level locations and augments the df_files that are created uniquely for each key in data_sources (the "experiment").

We might also want different behavior than CESM. For instance, if the data_sources is empty, maybe the default is to compile everything, whereas if data_sources is populated, only those specs are considered. This would be facilitated by promoting locations.

[matt-long]

closed by mistake.

[andersy005]

I think we should consider sticking with the CESM rubric

I like this approach. I am going to start working on it and will get back to you when I have questions for you :)

[andersy005]

Addressed in #43, #45