-
Notifications
You must be signed in to change notification settings - Fork 194
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ANN tests: make the min_recall check strict #1156
ANN tests: make the min_recall check strict #1156
Conversation
…rence against expected recall is not big
NB: some of the affected algorithms are not deterministic due to concurrency etc.; I've successfully run the test suite on my workstation 35 times, but we should keep an eye on how it behaves on the CI machines as well. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Artem for the PR! Indeed it is high time to enable tighter accuracy bounds. The PR looks good to me.
Codecov ReportBase: 87.99% // Head: 87.99% // No change to project coverage 👍
Additional details and impacted files@@ Coverage Diff @@
## branch-23.02 #1156 +/- ##
=============================================
Coverage 87.99% 87.99%
=============================================
Files 21 21
Lines 483 483
=============================================
Hits 425 425
Misses 58 58 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
My recent merge commit broke the build. Just leaving a note here that I'll fix this. |
Did a dozen tests on a100 and v100, seems to be passing with the last adjustments. |
/merge |
In #1135, we adjusted the min_recall values to report if any regressions happen in ivf-pq. However,
eval_neighbours
function, which is used in several ANN test suites, doesn't fail unless the regression is really large (it prints a warning if the calculated recall is "slightly" smaller than the expected recall).In this PR, I make
eval_neighbours
always fail if the calculated recall is smaller than the expected recall. Slightly adjust the tests and do a small refactoring along the way.