aaron's commit from refactoring PR

[bradyrx]

@aaronspring, this is a record of the second commit you made to #98. I wanted to have all of the changes for you here to cherry pick, unless you have them on your local update_dims branch. You can just open a new PR and apply these changes to address relative entropy/bootstrapping issues, etc.

[aaronspring]

• _m2m in bootstrap_perfect_model 3D, 1D
• implemented results as output from bootstrap_perfect_model
• control run doesn't contain all of the init years
  Comment: changed testing control
• bootstrap_perfect_model breaks with Datasets.
• Because of the following line in bootstrap_perfect_model, it breaks when any PM metrics are used...
  Comment: only test bootstrap_perfect_model for xskillscore_metrics for now until compute_persistence_pm can take all metrics
• change dims for rel_ent
• dummy data for rel_ent to test with treon (dont know how to properly setup treon but works locally)
• pytest on synthetic small data
• travis fails because it cant install bottleneck (https://github.com/bradyrx/climpred/pull/106)
• adapted PM notebook

Open questions:

• What do we actually want to test?
  • We mostly have acceptance tests (https://en.wikipedia.org/wiki/Acceptance_testing) that prove that they run without error and dont give nan results.
  • we only have very few unit tests
• Loading sample_data to test takes too long:
  123 passed, 48 warnings in 193.26 seconds
  • only test 1d ds and da, since additional coords should just go along?
  • restrict lon, lats even more

My vision for bootstrap_perfect_model:

• now implemented without caring much about control logic (Checking whether something is True or False)
• now by default arguments, I calcs skill, probability that persistence and uninit beat init and confidence intervals.
• especially CI takes a while because of the quantile function. Also because if comparison with persistence might not be needed, I have a flag to leave that out. performance-wise. lets discuss this in the next call.
• output:

results.coords
Coordinates:
  * i        (i) object 'init' 'pers' 'uninit'
    lon      (y, x) float64 161.9 163.6 165.2 166.9 ... -128.7 -127.1 -125.6
    lat      (y, x) float64 14.14 13.96 13.76 13.56 ... -32.44 -32.58 -32.72
  * lead     (lead) int64 1 2 3 4 5
  * results  (results) object 'skill' 'p' 0.025 0.975

• I still have to rename i. thats the dimension where you can choose among the different forecasts to look at results (skill, p and CI). Hope the output makes sense to you. I like that it even works for multiple variables in a ds.
  Maybe be the CIs should just be called lowCI highCI?

[bradyrx]

Just updated the Travis stuff so it should work properly in https://github.com/bradyrx/climpred/pull/106. It's now merged into master. So you should be able to run the following on your command line to get Travis to work:

git checkout master
git pull --rebase
git checkout aaron_bootstrap_updates
git rebase master
git push origin aaron_bootstrap_updates

I'll review this once you're done and request it.

[bradyrx]

What do we actually want to test?

We mostly have acceptance tests (https://en.wikipedia.org/wiki/Acceptance_testing) that prove that they run without error and dont give nan results. We only have very few unit tests

I should read up on testing. Acceptance testing seems like a necessity for all main functions as a first pass that they don't break (that's how we discovered that certain comparisons didn't work). Is there a way to smartly test that the results are accurate? I.e., can we have a few basic 1D skill assessments, save out the output as .npy files, and then in testing load them in and compare the results from the functions? So the two tests would be : (1) the function doesn't break + works with Datasets/DataArrays and (2) the function results are accurate.

Loading sample_data to test takes too long:
123 passed, 48 warnings in 193.26 seconds
only test 1d ds and da, since additional coords should just go along? restrict lon, lats even more

I agree. I did that just to make it work for now. I think we could just generate dummy data. But we can use a specific random seed to ensure the results are always the same and thus address point (2) from above.

[aaronspring]

I just self-reviewed because you started this PR. But I am waiting for you to have a look over it.
I updated the list of changes in comment 2.

Further changes needed:

• implement treon EDIT: fixed

Next PR:

• more clear strategy on what bootstrap_compute_perfect_model should be doing
• implement normalized (PM) metrics for persistence forecast

[bradyrx]

Awesome. I'll get back to this in a couple hours -- need to finish grading some homeworks.

[aaronspring]

CI breaks because of proplot:
ERROR in testing notebooks/object_oriented_demo.ipynb

[bradyrx]

CI breaks because of proplot:
ERROR in testing notebooks/object_oriented_demo.ipynb

Updated setup to include a pip install climpred['graphics'] option that installs proplot and matplotlib. Will see if that passes Travis while I'm looking through the rest of the PR.

[bradyrx]

object-oriented and relative-entropy notebooks are failing on me. Will check on those now.. doesn't look like anything too difficult.

[bradyrx]

Okay finally. Travis CI now works with treon and pytest. Fixed notebooks breaking, etc. Merging this now. Thanks for cleaning this up! Will chat on the phone about next crucial PRs.