Check if there is already old models that have been evaluated Fixes #540

[avishekrk]

When rerunning an experiment. This patch checks if the model_id has an entry in the evaluations table. If so, then it skips the evaluation saving time on longer model runs.

NOTE: If the type of evaluation parameters are different than the original model run the evaluation will still be skipped.

[jesteria]

And, BTW, the commit is messaged, "non-functioning check;" and, I'm not clear how this resolves #540…?

[thcrock]

Per @jesteria 's comment, this doesn't appear to be complete. As it stands, besides a test, there are some changes that I think are important even to what is being sketched out:

1. I do think we need to be sensitive to what evaluations are there. If we add a new evaluation metric, this should see it. We can probably make this check as coarse as: grab all of the metric/parameter combinations from the database for that model/date/as-of-date-frequency and compare with what is configured in the Evaluator. If not everything is there, re-do all of the evaluations.

2. The bulk of the logic should be in the ModelEvaluator class, as that's where a lot of the related data lives already. It should expose something like a needs_evaluation method that takes in a model and a matrix_store

3. This needs_evaluation should be called from the inner loop (where we actually call predict and evaluate).

[codecov-io]

Codecov Report

Merging #541 into master will increase coverage by 0.05%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #541      +/-   ##
==========================================
+ Coverage   82.63%   82.69%   +0.05%     
==========================================
  Files          82       82              
  Lines        4636     4651      +15     
==========================================
+ Hits         3831     3846      +15     
  Misses        805      805

Impacted Files	Coverage Δ
src/triage/component/catwalk/evaluation.py	97.41% <100%> (+0.29%)	⬆️
src/triage/component/catwalk/model_testers.py	93.54% <100%> (+0.69%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b89fce6...9a16646. Read the comment docs.

[thcrock]

I added some test cases for a ModelEvaluator.needs_evaluations method to this branch that can serve as a guide to finishing this feature