Checklist for v0.2 tag

[xylar]

To do list for MPAS-Analysis v. 0.2 and subsequent ACME analysis update:

• @pwolfram tests if Add autoclose argument to xarray.open_mfdataset call #151 works with the branch of xarray he's working to have in the next version
• @milenaveneziani or @pwolfram merges Add autoclose argument to xarray.open_mfdataset call #151 into MPAS-Analysis/develop
• @milenaveneziani merges Add support for short-term archiving #153
• @milenaveneziani merges Update start and end dates from climatology data set; issue a warning #157
• @pwolfram merges Update LANL configs, add new config for ACME B case with land-ice cavities #163 (which will be helpful for testing runs on LANL machines)
• @xylar merges Fixes dask out of memory error (sets max chunks) #166
• @xylar creates a test branch MPAS-Analysis/master_v0.2 where he merges MPAS-Analysis/develop into MPAS-Analysis/master
• @xylar updates the MPAS-Analysis submodule in PreAndPostProcessingScripts/xylar/update_MPAS_Analysis_v0.2 to point to the test branch
• Other updates in PreAndPostProcessingScripts/xylar/update_MPAS_Analysis_v0.2 for v0.2 support:
  • update variables in run*.csh scripts (@xylar)
  • update variables in python/setup_ocnice_config.py (@xylar)
  • update html generation (@milenaveneziani)
• Updates to observations, mapping and mask files:
  • Add mapping files for EC60to30v3_to_0.5x0.5degree on all machines (@milenaveneziani)
    • edison
    • lanl
    • aims
    • rhea
  • Add MOC region/transect files for all supported grids (@xylar) and move them to all machines
    • oEC60to30v1, oEC60to30v3, oEC60to30wLIv1, oQU240v1, oRRS18to6v1
    • edison
    • lanl
    • aims (@milenaveneziani)
    • rhea (@milenaveneziani)
• Preliminary testing on:
  • edison (@xylar)
    • beta0
    • beta1
  • lanl (@xylar)
• Coordinate with Workflow team to update anaconda on various machines:
  • establish common list of anaconda 2.7 modules
  • establish list of machines
  • establish maintainers for anaconda on each machine
  • get current package list of each machine
  • update edison (@xylar)
  • update cori (@xylar)
  • update lanl (@xylar)
  • update acme1 (@williams13)
  • update aims (@milenaveneziani)
  • update rhea / titan (@pwolfram)
• Test PreAndPostProcessingScripts/xylar/update_MPAS_Analysis_v0.2 on:
  • edison (@xylar)
  • cori (@xylar if there are any model results on Cori to test...)
  • lanl (@xylar)
  • aims (@milenaveneziani)
  • rhea (@milenaveneziani)
• @xylar pushes our test branch to MPAS-Analysis/master and creates a v0.2 tag. (I don't think this will change the commit for the submodule in PreAndPostProcessingScripts/xylar/update_MPAS_Analysis_v0.2, so no update should be needed on the ACME side, but @xylar will make sure.
• @xylar makes a PR for PreAndPostProcessingScripts/xylar/update_MPAS_Analysis_v0.2. (@kevans32 and @salilmahajan will review; @kevans32 will merge)

[milenaveneziani]

@pwolfram: do I have permission to update conda on rhea?

[pwolfram]

@milenaveneziani, I don't think so. I'll update conda once xarray is pushed to the minor version number by @shoyer.

[xylar]

@milenaveneziani, if you can point me to a run with the EC60to30v3 grid, I can make mapping files and region masks for that. Then, I should be able to replace the old region mask for the v1 grid (with the less accurate southern transect) with an updated version at the same time I put in a region mask file and a mapping file for the v3 grid.

[xylar]

@milenaveneziani, there's an issue with the index.html not pointing to the right file names if the climatology (or, in many cases, time series) did not include all of the requested years. I don't know how the code for creating the html file can know what the actual range of available output was to correct for this problem. Perhaps we say for now that the html doesn't work unless you make sure to set the start and end years to be within the range of the actual results?

[milenaveneziani]

that's a good point @xylar.. There is another csh script that makes a case.tmp file, storing env variables in it, including the begin_yr and end_yr values that are set in run_*.csh. So, yes, those will be the intended years, so the real ones.
One option could be to test, in run_*.csh, that the choices of begin_yr,end_yr are appropriate for the particular data set in question, and then issue an error if they are not. In fact, I don't know if the atmosphere diagnostics support an out-of-range case at all, so that would be a good idea in general for the ACME diags.

[xylar]

@milenaveneziani, feel free to add such a test and push it to my ACME analysis branch.

[xylar]

To test on NERSC with the updated anaconda configuration, please modify the appropriate line of your custom run_EDISON.csh to be:

module load python/anaconda-2.7-climate_test

Essentially simultaneously with the merge of https://github.com/ACME-Climate/PreAndPostProcessingScripts/pull/29, I will update the shared anaconda environment to match this one.

[milenaveneziani]

@xylar: I am thinking about adding a simple test for this, considering that the cshell scripts will be changing/disappearing in the near future anyway. I would parse in the year value that is in a rpointer file: that would be part of the restart date for the next restart cycle, thus indicating that data is currently available for years<rpointer_date. That of course will not mean that all files will be available between year=1 and year=rpointer_date-1, but it seems like the easiest way to do this in a temporary manner.
What do you think?

[xylar]

@milenaveneziani, So if I understand right, you would take the values read in but change the end year if it's beyond the date in the restart pointer? That sounds reasonable to me. Eventually, we'll probably want to just parse it out of the file names or the output config files themselves but that's much easier to do in python.

[milenaveneziani]

If end_yr_climo or end_yr_ts are greater than the year in the rpointer file, I was actually just thinking to issue a warning and exit the cshell script all together.
This is because I am not sure the atm diagnostics can even handle the case when end_yr is greater than the max available year.
We can improve when everything is in python, of course.

[xylar]

Ah, right. You said that was the plan and I just forgot.