Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(types): add ranking evaluation mixin #3698

Merged
merged 13 commits into from
Nov 13, 2021
Merged

feat(types): add ranking evaluation mixin #3698

merged 13 commits into from
Nov 13, 2021

Conversation

JoanFM
Copy link
Member

@JoanFM JoanFM commented Oct 18, 2021 •
edited

No description provided.

@JoanFM JoanFM linked an issue Oct 18, 2021 that may be closed by this pull request
Add evaluation modules to DocumentArrayLike #3695
Closed
@github-actions github-actions bot added size/L area/core This issue/PR affects the core codebase area/testing This issue/PR affects testing component/type labels Oct 18, 2021
@codecov
Copy link

codecov bot commented Oct 18, 2021 •
edited

Codecov Report

Merging #3698 (b868847) into master (62f244f) will increase coverage by 0.68%.
The diff coverage is 89.56%.

❗ Current head b868847 differs from pull request most recent head 8d0284e. Consider uploading reports for the commit 8d0284e to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3698      +/-   ##
==========================================
+ Coverage   87.76%   88.45%   +0.68%     
==========================================
  Files         174      176       +2     
  Lines       12285    12376      +91     
==========================================
+ Hits        10782    10947     +165     
+ Misses       1503     1429      -74
Flag Coverage Δ
daemon 43.70% <28.69%> (-0.23%) ⬇️
jina 85.44% <89.56%> (+1.58%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jina/types/arrays/mixins/magic.py 95.00% <ø> (-0.24%) ⬇️
jina/types/arrays/mixins/traverse.py 90.62% <0.00%> (-1.57%) ⬇️
jina/types/document/mixins/helper.py 72.54% <ø> (-1.04%) ⬇️
jina/types/document/mixins/plot.py 91.52% <ø> (ø)
jina/types/score/map.py 96.77% <50.00%> (ø)
jina/types/arrays/mixins/evaluation.py 80.00% <80.00%> (ø)
jina/math/evaluation.py 92.85% <92.85%> (ø)
jina/helper.py 81.60% <100.00%> (+0.21%) ⬆️
jina/types/arrays/mixins/__init__.py 100.00% <100.00%> (ø)
jina/types/arrays/mixins/empty.py 100.00% <100.00%> (+18.18%) ⬆️
... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62f244f...8d0284e. Read the comment docs.

@github-actions
Copy link

github-actions bot commented Oct 18, 2021 •
edited

Latency summary

Current PR yields:

  • 😶 index QPS at 1267, delta to last 2 avg.: -2%
  • 😶 query QPS at 55, delta to last 2 avg.: -2%
  • 😶 dam extend QPS at 36692, delta to last 2 avg.: -1%
  • 😶 avg flow time within 1.6093 seconds, delta to last 2 avg.: +2%
  • 🐢🐢 import jina within 0.4152 seconds, delta to last 2 avg.: -10%

Breakdown

Version Index QPS Query QPS DAM Extend QPS Avg Flow Time (s) Import Time (s)
current 1267 55 36692 1.6093 0.4152
2.4.3 1269 55 36128 1.5655 0.4279
2.4.2 1337 57 38371 1.574 0.4948

Backed by latency-tracking. Further commits will update this comment.

@JoanFM JoanFM changed the title Ranking evaluation mixin feat(types): add ranking evaluation functionality Oct 18, 2021
@JoanFM JoanFM requested review from maximilianwerk, hanxiao and a team October 18, 2021 14:48
@JoanFM JoanFM marked this pull request as ready for review October 19, 2021 09:23
@github-actions github-actions bot added the area/docs This issue/PR affects the docs label Oct 19, 2021
hanxiao
hanxiao requested changes Oct 19, 2021
View reviewed changes
Copy link
Member

@hanxiao hanxiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please wait until i have time to review it, probably on Thursday

@JoanFM JoanFM changed the title feat(types): add ranking evaluation functionality feat(types): add ranking evaluation mixin Nov 12, 2021
@hanxiao
Merge branch 'master' into ranking-evaluation-mixin
27d4560 
# Conflicts:
#	docs/fundamentals/document/documentarray-api.md
@hanxiao hanxiao marked this pull request as draft November 13, 2021 15:36
@github-actions github-actions bot added size/M area/helper This issue/PR affects the helper functionality component/math and removed area/docs This issue/PR affects the docs labels Nov 13, 2021
@github-actions
Copy link

📝 Docs are deployed on https://ranking-evaluation-mixin--jina-docs.netlify.app 🎉

@hanxiao hanxiao marked this pull request as ready for review November 13, 2021 17:13
@hanxiao hanxiao merged commit 5915342 into master Nov 13, 2021
@hanxiao hanxiao deleted the ranking-evaluation-mixin branch November 13, 2021 17:40
@github-actions github-actions bot added area/cicd This issue/PR affects the cicd pipeline area/docs This issue/PR affects the docs area/housekeeping This issue/PR is housekeeping labels Nov 13, 2021
JoanFM
JoanFM commented Nov 13, 2021
View reviewed changes
tests/unit/types/arrays/mixins/test_eval_class.py
('ndcg_at_k', {}),
],
)
def test_eval_mixin_perfect_match(metric_fn, kwargs):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hanxiao I do not understand how is here NDCG evaluated? How can I pass relevance information of the different groundtruth elements?

tests/unit/types/arrays/mixins/test_eval_metrics.py
def test_ndcg_at_k():
NDCG_K_VAL = 1.0

relevance = [3, 2, 3, 0, 0, 1, 2, 2, 3, 0]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this supposed to be passed through the DA.evaluate method?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hanxiao hanxiao hanxiao requested changes

@maximilianwerk maximilianwerk Awaiting requested review from maximilianwerk

Assignees
No one assigned
Labels
area/cicd This issue/PR affects the cicd pipeline area/core This issue/PR affects the core codebase area/docs This issue/PR affects the docs area/helper This issue/PR affects the helper functionality area/housekeeping This issue/PR is housekeeping area/testing This issue/PR affects testing component/math component/type size/L size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add evaluation modules to DocumentArrayLike
2 participants
@JoanFM @hanxiao