Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor existing functionality to make intake-esm robust and extensible #77

Merged
merged 21 commits into from
Jun 14, 2019
Merged

Refactor existing functionality to make intake-esm robust and extensible #77

merged 21 commits into from
Jun 14, 2019

Conversation

andersy005
Copy link
Member

@andersy005 andersy005 commented Jun 11, 2019

Notable Changes

  • Partially addresses Needed Code Refactoring  #70 by cleaning up test directory.

  • Closes example_input is out of date #73

  • Building CMIP5 and CMIP6 doesn't rely on directory structure anymore. Just like CESM data holdings, attributes are extracted from the filename, and any additional attributes (e.g. grid_label, version) are extracted from the file fullpath.

  • All collections inherit from Collection base class located in intake_esm/collection.py.

  • The structure of the input YAML file is now consistent across all collections.

Example snippets:

  • CMIP6
name: cmip6_test_collection
collection_type: cmip6
data_sources:
 BCC-CSM2-MR:
   locations:
    -  name: SAMPLE-DATA
       loc_type: posix
       direct_access: True
       urlpath: ./tests/sample_data/cmip/CMIP6/CMIP/BCC/BCC-CSM2-MR
       exclude_dirs: ['*/files/*', 'latest']
       file_extension: .nc

   extra_attributes:
     mip_era: CMIP6
     activity_id: CMIP
     institution_id: BCC

 BCC-ESM1:
   locations:
    -  name: SAMPLE-DATA
       loc_type: posix
       direct_access: True
       urlpath: ./tests/sample_data/cmip/CMIP6/CMIP/BCC/BCC-ESM1
       exclude_dirs: ['*/files/*', 'latest']
       file_extension: .nc

   extra_attributes:
     mip_era: CMIP6
     activity_id: CMIP
     institution_id: BCC
  • CESM
name: cesm1-le
collection_type: cesm
data_sources:
  CTRL:
    locations:
      - name: SAMPLE-DATA
        loc_type: posix
        direct_access: True
        urlpath: ./tests/sample_data/cesm-le
        file_extension: .nc
    extra_attributes:
      component_attrs:
        ocn:
          grid: POP_gx1v6
      case_members:
        - case: b.e11.B1850C5CN.f09_g16.005
          sequence_order: 0
          ensemble: 5
          has_ocean_bgc: True
          year_offset: 1448
  20C:
    locations:
      - name: SAMPLE-DATA
        loc_type: posix
        direct_access: True
        urlpath: ./tests/sample_data/cesm-le
        file_extension: .nc
    extra_attributes:
      component_attrs:
        ocn:
          grid: POP_gx1v6
      case_members:
        - case: b.e11.B20TRC5CNBDRD.f09_g16.001
          sequence_order: 0
          ensemble: 1
          has_ocean_bgc: True

@matt-long, your review/comments would be helpful prior merging this into master.

@andersy005 andersy005 changed the title First stab at refactoring existing functionality Refactor existing functionality to make intake-esm robust and extensible Jun 11, 2019
@andersy005 andersy005 marked this pull request as ready for review June 13, 2019 01:21
@andersy005 andersy005 added this to In progress in Backlog via automation Jun 13, 2019
@andersy005 andersy005 merged commit 830fa12 into intake:master Jun 14, 2019
Backlog automation moved this from In progress to Done Jun 14, 2019
This was referenced Jun 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Backlog
  
Done
Development

Successfully merging this pull request may close these issues.

example_input is out of date
1 participant