Fix binary classification pipeline thresholding

[bchen1116]

fix #3086

Perf tests here
report_threshold.html.zip

Tests for two other methods here:
report_threshold_binexp.html.zip
report_threshold_randexp.html.zip

Current defaults are faster, so we will continue with that.

Writeup under name "Global Binary Pipeline Thresholding Perf Tests"

[codecov]

Codecov Report

Merging #3280 (bc2d466) into main (a18efa8) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3280     +/-   ##
=======================================
+ Coverage   99.8%   99.8%   +0.1%     
=======================================
  Files        322     322             
  Lines      31614   31630     +16     
=======================================
+ Hits       31524   31540     +16     
  Misses        90      90             

Impacted Files	Coverage Δ
...alml/objectives/binary_classification_objective.py	100.0% <100.0%> (ø)
...odel_understanding_tests/test_decision_boundary.py	100.0% <100.0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a18efa8...bc2d466. Read the comment docs.

[bchen1116]

Change needed to get the assert to pass in this file. Likely due to the new thresholding method

[freddyaboulton]

I'm confused by this. If this method finds the global max, and the other found a local max, shouldn't this method give the same score or higher with the same parameters?

[bchen1116]

this method is a global minimum optimizer, whereas minimize_scalar was a local minimizer. This will take a little longer, but should allow us to avoid most issues where the optimized threshold doesn't align with our expectations

[chukarsten]

Is this supported by perf tests? Would be nice to know before going into the release next week what to expect.

[chukarsten]

Feel free to run n_trials=1 if you do them

[bchen1116]

Ran them yesterday, will post results shortly!

[jeremyliweishih]

Looks great to me!

[jeremyliweishih]

might be interesting to play with some of the parameters to see if we can get the same performance but quicker. Def not blocking though!

[bchen1116]

From my impression, the default values should perform fairly similar in speed to the other optimization methods available to this. I can try running a few other perf tests to double check though!

[freddyaboulton]

@bchen1116 I'm confused as to why we need to change the parameters in the lead scoring doc. Doesn't that mean this method found a threshold that's less optimas than the previous method?

[freddyaboulton]

I'm confused by this. If this method finds the global max, and the other found a local max, shouldn't this method give the same score or higher with the same parameters?

[bchen1116]

@freddyaboulton, took me a while to track down why the scores changed.
First, the thresholding data was optimized on a subset of the training data, while the holdout data is used to score. This can lead to deviations in holdout data performance. Along with that, here are two screenshots of the thresholding parameter and associated objective score:

Threhsolding param and Lead scoring value [This branch] (I lightly highlighted the threshold and score of the AUC/F1 oriented run):

Thresholding param and F1 value [This branch]:

Thresholding param and Lead scoring [Main branch]:

I've highlighted the optimal objectives here for the optimal thresholds on this new branch. Using the current parameters, the F1 score is highest at a threshold of ~0.44, while the lead scoring is highest at a threshold of ~0.40. Because of this slight deviation, we see a drop in the actual lead scoring value on the holdout data, which is why I needed to change the lead scoring objective parameters.

We see that on the main branch, we don't even search for objective scores when the threshold is below 0.42. This would explain why we see such a difference, since the local optimization method on main doesn't search as thoroughly as the global method here. While it does find a local minima, we don't find as optimal of a minima as with this new branch. Although this leads to a lower holdout score on this new branch, that is likely due to the way we structured the Lead scoring objective cost function itself.

I believe it's fine to move forward with this PR

[freddyaboulton]

Thanks for the explanation @bchen1116 !

[chukarsten]

LGTM!