Add recommendation breakdown function - #4188
Conversation
Codecov Report
@@ Coverage Diff @@
## main #4188 +/- ##
=======================================
+ Coverage 99.7% 99.7% +0.1%
=======================================
Files 349 349
Lines 38107 38128 +21
=======================================
+ Hits 37990 38011 +21
Misses 117 117
|
jeremyliweishih
left a comment
jeremyliweishih
left a comment
There was a problem hiding this comment.
LGTM but wondering if we should include the score weights as well!
| rankings_df.reset_index(drop=True, inplace=True) | ||
| return rankings_df | ||
|
|
||
| def get_recommendation_score_breakdown(self, pipeline_id): |
There was a problem hiding this comment.
should we include the score weights as well?
There was a problem hiding this comment.
I'm leaning towards not - the score weights are known and/or set ahead of time, while the scores themselves are not. I'd rather keep this simpler and more understandable, but I am open to adding something if you disagree!
There was a problem hiding this comment.
where do we display the weights to the user? I don't see anything here so far. If the purpose of get_recommendation_score_breakdown is to explain to the user how the recommendation score is calculated, I think we should give them the weights so they know how they go from the metrics to the recommendation score.
There was a problem hiding this comment.
The weights aren't directly displayed to the user, but the docstring states that they are weighted equally by default and we offer methods for customization by the user.
The goal of this function isn't to explicitly explain to the user how the recommendation score was calculated - that's why the values reported are explicitly the non-normalized ones. Rather, the goal is to present the user with the evaluated objectives behind the recommendation score. What they do with that information is up to them, but it's more to give them the insight about direct model performance rather than "here's the equation".
chukarsten
left a comment
chukarsten
left a comment
There was a problem hiding this comment.
This looks good and looks like it will play nicely with the further stories to update this functionality. Left a nit about parametrizing the test, but other than that, looks good! Accept or reject my suggestion as you see fit.
3 participants
Closes #4187