Save computed additional objectives during search

[eccabay]

Closes #4140

[codecov]

Codecov Report

Merging #4141 (c50c229) into main (7bc6210) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #4141     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        349     349             
  Lines      37770   37778      +8     
=======================================
+ Hits       37653   37661      +8     
  Misses       117     117             

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	99.6% <100.0%> (+0.1%)	⬆️
evalml/tests/automl_tests/test_automl.py	99.6% <100.0%> (+0.1%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[jeremyliweishih]

can you clarify for me why we take the holdout score for ranking_additional_objectives here?

[eccabay]

The end goal here is to always use the holdout scores instead of the mean cv scores, since it's conceptually a better method for measuring performance. However, I wanted to keep this as flexible as possible, so that we can still have access to results even if the holdout isn't run, as is the current default. This will enable us to build a recommendation score faster, regardless of the holdout score work.

[jeremyliweishih]

LGTM