Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ranking Metrics - Better Precision/Recall/MRR calculation #1492

Merged
merged 11 commits into from Mar 26, 2024
Merged

Ranking Metrics - Better Precision/Recall/MRR calculation #1492

merged 11 commits into from Mar 26, 2024

Conversation

FelipeAdachi
Copy link
Contributor

@FelipeAdachi FelipeAdachi commented Mar 20, 2024 •
edited

Description

This PR:

  • Makes the metrics calculation (other than NDCG and AP, which are already correct) correct for boolean and numeric scenarios
  • Changes ndcg from a non-mergeable batch metric to a mergeable row-wise metric
  • Corrects top_rank to return None instead of 0 when there is no relevant item (since lower is better for top_rank, assigning a 0 is not correct)
  • Changes MRR from a non-mergeable batch metric to a mergeable row-wise metric (and renamed from Mean Reciprocal Rank to Reciprocal Rank)
  • Refactors the code to group row-wise statistics calculations functions

After these changes, the only metric that makes the ranking metrics non-mergeable is accuracy_k

felipe207 added 10 commits March 18, 2024 18:06
ranking example rewrite
53913eb
Merge branch 'mainline' into dev/felipe/ranking-example
ddc7faf 
# Conflicts:
#	python/examples/experimental/Writing_Ranking_Performance_Metrics_to_WhyLabs.ipynb
rewrite
53c8059
wip
35abcf4
wip
8721056
fix other metrics
754327b
Merge branch 'mainline' into dev/felipe/precision-recall-ranking
d380cf8 
# Conflicts:
#	python/examples/experimental/Writing_Ranking_Performance_Metrics_to_WhyLabs.ipynb
#	python/whylogs/experimental/api/logger/__init__.py
update notebook results
4889a66
add tests for precision and recall at k
f32a356
fix several metrics
e7d8fe0
richard-rogers
richard-rogers reviewed Mar 25, 2024
View reviewed changes
Copy link
Contributor

@richard-rogers richard-rogers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Around line 143 in __init__.py a doc string explaining when to use which arguments would be helpful.

jamie256
jamie256 approved these changes Mar 26, 2024
View reviewed changes
Copy link
Contributor

@jamie256 jamie256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jamie256 jamie256 merged commit 538eb05 into mainline Mar 26, 2024
19 checks passed
@jamie256 jamie256 deleted the dev/felipe/precision-recall-ranking branch March 26, 2024 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richard-rogers richard-rogers richard-rogers left review comments

@jamie256 jamie256 jamie256 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@FelipeAdachi @jamie256 @richard-rogers