Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix sparse search #62

Merged
merged 2 commits into from
Feb 16, 2022
Merged

Fix sparse search #62

merged 2 commits into from
Feb 16, 2022

Conversation

maximedb
Copy link
Contributor

@maximedb maximedb commented Feb 9, 2022

Hi,

This pull request fixes two issues with the sparse search:

  1. Because of the step in trange, some queries are never evaluated (only the first one of each batch is evaluated) as reported in Error in beir/beir/retrieval/search/sparse/sparse_search.py #60.
  2. Dense search skips some results (see here), this was not implemented in the sparse search. It impacts the results on ArguAna and Quora (Can't understand a certain line #37)

Let me know if you have any questions.

Maxime.

@maximedb maximedb mentioned this pull request Feb 9, 2022
@thakur-nandan
Copy link
Member

Hi @maximedb and @jin530,

Thanks for the pull request. Indeed as mentioned in #60, this is an error that sadly I wasn't able to find earlier myself.
Nevertheless, the scores for SPARTA reported in the BEIR paper are accurate. It used an old script that did not have the issue. I will soon have a look and merge the PR.

Kind Regards,
Nandan Thakur

@nreimers nreimers merged commit a55552d into beir-cellar:main Feb 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants