Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: Adapt retriever benchmarks script #5004

Merged
merged 13 commits into from
May 25, 2023
Merged

refactor: Adapt retriever benchmarks script #5004

merged 13 commits into from
May 25, 2023

Conversation

bogdankostic
Copy link
Contributor

@bogdankostic bogdankostic commented May 23, 2023 •
edited
Loading

Related Issues

Related PRs

Proposed Changes:

This PR adapts the script for benchmarking retriever-only pipelines. Indexing and querying are benchmarked separately.

How did you test it?

Manual tests.

Notes for the reviewer

We should use run_batch instead of eval_batch methods as measuring the time of eval isn’t representative for pure inference speed because it creates a significant overhead.

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added tests that demonstrate the correct behavior of the change
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

This was referenced May 23, 2023
Closed
Merged
Merged
@coveralls
Copy link
Collaborator

coveralls commented May 23, 2023 •
edited
Loading

Pull Request Test Coverage Report for Build 5080178368

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 39.621%
Totals Coverage Status
Change from base Build 5079928368: 0.0%
Covered Lines: 8876
Relevant Lines: 22402
💛 - Coveralls

@bogdankostic bogdankostic marked this pull request as ready for review May 25, 2023 09:20
@bogdankostic bogdankostic requested a review from a team as a code owner May 25, 2023 09:20
@bogdankostic bogdankostic requested review from masci and vblagoje and removed request for a team May 25, 2023 09:20
vblagoje
vblagoje approved these changes May 25, 2023
View reviewed changes
Copy link
Member

@vblagoje vblagoje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@bogdankostic bogdankostic merged commit 6e10fda into main May 25, 2023
44 checks passed
@bogdankostic bogdankostic deleted the adapt_retriever_benchmarks branch May 25, 2023 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vblagoje vblagoje vblagoje approved these changes

@masci masci Awaiting requested review from masci masci is a code owner automatically assigned from deepset-ai/open-source-engineering

Assignees
No one assigned
Labels
topic:tests type:documentation Improvements on the docs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@bogdankostic @coveralls @vblagoje