Remove unused fit_needs_proba parameter from objective class

[dsherry]

I don't see this parameter used anywhere in our codebase and would like to delete it

---

After creating the pull request: in order to pass the changelog_updated check you will need to update the "Future Release" section of docs/source/changelog.rst to include this pull request by adding :pr:123.

[codecov]

Codecov Report

Merging #320 into master will increase coverage by 0.09%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #320      +/-   ##
==========================================
+ Coverage   97.04%   97.14%   +0.09%     
==========================================
  Files          96       96              
  Lines        2982     2974       -8     
==========================================
- Hits         2894     2889       -5     
+ Misses         88       85       -3

Impacted Files	Coverage Δ
evalml/objectives/fraud_cost.py	100% <ø> (ø)	⬆️
evalml/objectives/lead_scoring.py	96.55% <ø> (-0.12%)	⬇️
evalml/objectives/objective_base.py	97.43% <ø> (-0.07%)	⬇️
evalml/pipelines/pipeline_base.py	98.6% <100%> (+1.97%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4874fc...3c09065. Read the comment docs.

[jeremyliweishih]

It is to my understanding that fit_needs_proba is for objectives that require predicted probabilities instead of predicted labels or values when fitting for the objective. The clearest example would be any objective that needs to be able to learn the best probability threshold for a binary classification objective. We currently do that for both our custom objectives (fraud and lead scoring) and I believe sklearn does it internally for binary classification objectives. If we were to standardize objective fitting to use probabilities instead of predicted labels or values it could break fitting objectives for any custom regression objective or any objective that needs fitting with predicted values or labels.

[dsherry]

@jeremyliweishih oh, interesting. So you're saying that this flag would be used by a custom objective for regression? My understanding after reading the code was that we don't support custom objectives for regression, but you're causing me to think again. So you'd set fit_needs_proba to False for a regression problem? I will reread the code with this in mind and see if I can summarize the problem I'm seeing

[jeremyliweishih]

To clarify: the flag is being used to denote whether an objective needs to fit using predicted probabilities or predicted labels/values. In the case of our two custom objectives, they both require the predicted probability but I can imagine in the future where a custom objective would need predicted labels/values instead.

[dsherry]

@jeremyliweishih right, I agree. This parameter would be useful to support a future feature where custom objectives can pass non-probabilities into decision_function and objective_function. However, I think there's enough rewriting of other stuff we'd need to do in order to get there (for example, decision_function is not applicable for regression), that the easiest thing to do is to remove this parameter right now, and add that functionality back in later when we have better abstractions in place. Does that seem reasonable?

[jeremyliweishih]

@dsherry

[kmax12]

good to merge, unless we want to hold off for full redesign.

[angela97lin]

LGTM, but as Max suggested, would it be better to wait until our design is settled on and fix it afterwards? :o

[dsherry]

Thanks for the reviews. I'm gonna merge this! Deleting code is one of my favorite things to do 😁🎊

To summarize / for future reference: my understanding of what fit_needs_proba does currently is it allows users to choose if they want to provide classification probabilities v.s. classified values to both a) the custom objective decision_function during both a) binary classifier threshold optimization (aka "fitting" and b) classifier prediction. I don't understand why that would be desirable. We decided to remove it because 1) it's unused and undocumented in our codebase, and it's difficult to articulate its value 2) if this parameter was intended to support custom objectives for regression, there's plenty of other things which need to change or be added to custom metrics in order to support that, 3) we'll be removing this as part of the objective API project anyways.