Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ANN tests: make the min_recall check strict #1156

Merged

Conversation

achirkin
Copy link
Contributor

In #1135, we adjusted the min_recall values to report if any regressions happen in ivf-pq. However, eval_neighbours function, which is used in several ANN test suites, doesn't fail unless the regression is really large (it prints a warning if the calculated recall is "slightly" smaller than the expected recall).
In this PR, I make eval_neighbours always fail if the calculated recall is smaller than the expected recall. Slightly adjust the tests and do a small refactoring along the way.

@achirkin achirkin requested a review from a team as a code owner January 19, 2023 12:08
@github-actions github-actions bot added the cpp label Jan 19, 2023
@achirkin achirkin added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jan 19, 2023
@achirkin achirkin requested a review from tfeher January 19, 2023 12:09
@achirkin
Copy link
Contributor Author

achirkin commented Jan 19, 2023

NB: some of the affected algorithms are not deterministic due to concurrency etc.; I've successfully run the test suite on my workstation 35 times, but we should keep an eye on how it behaves on the CI machines as well.

Copy link
Contributor

@tfeher tfeher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Artem for the PR! Indeed it is high time to enable tighter accuracy bounds. The PR looks good to me.

@codecov-commenter
Copy link

codecov-commenter commented Jan 21, 2023

Codecov Report

Base: 87.99% // Head: 87.99% // No change to project coverage 👍

Coverage data is based on head (d317b6e) compared to base (0e96662).
Patch has no changes to coverable lines.

Additional details and impacted files
@@              Coverage Diff              @@
##           branch-23.02    #1156   +/-   ##
=============================================
  Coverage         87.99%   87.99%           
=============================================
  Files                21       21           
  Lines               483      483           
=============================================
  Hits                425      425           
  Misses               58       58           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@cjnolet
Copy link
Member

cjnolet commented Jan 22, 2023

My recent merge commit broke the build. Just leaving a note here that I'll fix this.

@achirkin
Copy link
Contributor Author

achirkin commented Jan 23, 2023

Did a dozen tests on a100 and v100, seems to be passing with the last adjustments.

@cjnolet
Copy link
Member

cjnolet commented Jan 23, 2023

/merge

@rapids-bot rapids-bot bot merged commit 5a6cb09 into rapidsai:branch-23.02 Jan 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Ready to Merge cpp improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
Development

Successfully merging this pull request may close these issues.

4 participants