Create testing suite of cases for main computation loop

[spencerahill]

Unit tests are badly needed pretty much everywhere. But in particular, a suite of test-cases should be implemented for the main Calc.compute that span all of the possible permutations of input and ouput specifications. Reason being: frequently when modifying the Calc class, out of laziness I just do an ad-hoc test using a single permutation of the myriad data input/output options...and then when I need to use a different combination, I come to find that the changes I implemented broke the code for this particular case.

A few in particular that come to mind (additions are welcome; I will update this as they come up):

Input

• Instantaneous v. time-series v. time-averaged
• Constant, 1D, 2D, 3D, and 4D input data
• Variables that are functions of other variables vs. standalone
• Variables whose functions use grid-data vs. do not
• Yearly interval: sub-annual v. whole year
• Date range: spanning single year or less v. multiple years
• Vertically defined data: on pressure v. hybrid coordinates
• Input files with non-standard coordinate names (e.g. "LATITUDE")
• Start date with year that's not the same as the start year of the netCDF file it comes from.
• Variable that is a function of other variables, where those dependency-variables come from different runs and/or models.

Output

• Vertically integrate and don't, attempted on both vertically defined and not vertically defined
• Functions that use xray resampling on time coordinate (e.g. time tendencies).
• ts, av, and std on lat-lon data
• ts, av, and std on regional averages

[spencerkclark]

One that comes to mind to add to the Input section is:

• Input files with non-standard coordinate names (e.g. "LATITUDE")

If you don't mind, in the future if something comes up I'll just add it to the list directly within your initial comment rather than have a long thread. I'm happy to help in writing some of these tests.

[spencerahill]

That's a good one...yes please feel free to just add to the list.

[spencerahill]

@spencerkclark how much of the above cases test_calc_basic cover (feel free to check them off). And how do we feel about the rest?

[spencerkclark]

@spencerahill thanks reminding me about this; I agree this needs to be updated.

Our operations involving time are pretty well tested at the moment -- all the methods in utils.times are well covered (and actually have unit tests dedicated just to them rather than implicitly through test_calc_basic.py). Therefore while we don't necessarily explicitly test

Start date with year that's not the same as the start year of the netCDF file it comes from.
Date range: spanning single year or less v. multiple years
Yearly interval: sub-annual v. whole year

I feel pretty confident that those cases will be handled correctly (just because nearly all the aospy-specific logic is tested here, and the rest gets pushed to upstream libraries).

test_calc_basic tests:

• 3D input data only (lat, lon, time) -- no tests involving vertically defined data are included
• both model native and aospy-computed variables
• regional averages, so the surface area grid attribute is used in a calculation
• only time average input data (monthly means)

That said, I much prefer the way things are tested for instance in utils.times (one method at a time). I think it gets complicated to try and test every possible case, starting from the Calc.compute step. What are your thoughts on this?

[spencerahill]

That said, I much prefer the way things are tested for instance in utils.times (one method at a time). I think it gets complicated to try and test every possible case, starting from the Calc.compute step. What are your thoughts on this?

I agree. I'd say we can add tests of specific Calc cases if/when bugs arise, but otherwise we should focus our testing efforts on more granular unit tests. So I'll close this.