Save computed additional objectives during search #4141
Conversation
Codecov Report
@@ Coverage Diff @@
## main #4141 +/- ##
=======================================
+ Coverage 99.7% 99.7% +0.1%
=======================================
Files 349 349
Lines 37770 37778 +8
=======================================
+ Hits 37653 37661 +8
Misses 117 117
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
eccabay
requested review from
jeremyliweishih,
christopherbunn and
chukarsten
and removed request for
jeremyliweishih
| else: | ||
| holdout_scores = evaluation_results["holdout_scores"] | ||
| ranking_additional_objectives = dict(holdout_scores) |
There was a problem hiding this comment.
can you clarify for me why we take the holdout score for ranking_additional_objectives here?
There was a problem hiding this comment.
The end goal here is to always use the holdout scores instead of the mean cv scores, since it's conceptually a better method for measuring performance. However, I wanted to keep this as flexible as possible, so that we can still have access to results even if the holdout isn't run, as is the current default. This will enable us to build a recommendation score faster, regardless of the holdout score work.
Closes #4140