Introduce the concept of ranking metrics #3721
Conversation
Codecov Report
@@ Coverage Diff @@
## main #3721 +/- ##
=======================================
+ Coverage 99.7% 99.7% +0.1%
=======================================
Files 339 339
Lines 34819 34845 +26
=======================================
+ Hits 34688 34714 +26
Misses 131 131
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
eccabay
requested review from
jeremyliweishih,
chukarsten,
christopherbunn and
fjlanasa
docs/source/demos/text_input.ipynb
Outdated
| @@ -453,7 +453,7 @@ | |||
| ], | |||
| "metadata": { | |||
| "kernelspec": { | |||
| "display_name": "Python 3", | |||
| "display_name": "Python 3 (ipykernel)", | |||
There was a problem hiding this comment.
Might need a make lint-fix?
| obj() | ||
| for obj in all_objectives_dict.values() | ||
| if obj.is_defined_for_problem_type(problem_type) | ||
| and (obj not in get_non_core_objectives() or obj in ranking_only_objectives()) |
There was a problem hiding this comment.
remind me what's behind the obj not in get_non_core_objectives()
There was a problem hiding this comment.
There are some objectives in that list that require additional setup/configuration before they can be used during optimization. An example, the cost benefit matrix needs weights for each true positive/negative and false positive/negative
| Returns: | ||
| List of ObjectiveBase classes | ||
| """ | ||
| return [ |
There was a problem hiding this comment.
thoughts on adding ranking_only as a property instead of maintaining a list? I know this is what we do for get_non_core_objectives as well but maybe we could file this as a future enhancement.
There was a problem hiding this comment.
I'm not quite sure I understand your suggestion - do you mean a property of the objective classes themselves? I do actually like that better, I think.
There was a problem hiding this comment.
Quick investigation shows that making this switch is unfortunately nontrivial, so I think I'll leave it as is for now. It's a good future enhancement though!
Closes #3710
Note: with this implementation,
get_optimization_objectivesis a roundabout way to get our current core metrics. We will want to move away from core for optimization purposes, but we can’t remove the non-optimization non-core (i.e. recall) metrics until our downstream users have made the switch.