Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: Adapt running benchmarks #5007

Merged
merged 31 commits into from
May 26, 2023
Merged

refactor: Adapt running benchmarks #5007

merged 31 commits into from
May 26, 2023

Conversation

bogdankostic
Copy link
Contributor

@bogdankostic bogdankostic commented May 23, 2023 •
edited
Loading

Related Issues

Related PRs

Proposed Changes:

This PR adapts the way benchmarks can be run.
run.py takes as input a config file and optionally an output path.
The config file is a Pipeline YAML file containing a querying pipeline and, in case the querying pipeline contains a retriever, an indexing pipeline. Additionally, beside the pipeline configuration, the config file contains a section benchmark_config where the following information should be specified:

  • labels_file: A SQuAD-formatted json or csv containing the labels to benchmark on.
  • documents_directory: A Path to a directory containing the files that should be indexed into the document store (Only needed for retriever and retriever-reader pipelines.)
  • data_url: Optionally. Allows to download data from this URL and save it in the directory data/.

How did you test it?

Manual tests.

Notes for the reviewer

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added tests that demonstrate the correct behavior of the change
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@bogdankostic bogdankostic mentioned this pull request May 23, 2023
Closed
@coveralls
Copy link
Collaborator

coveralls commented May 23, 2023 •
edited
Loading

Pull Request Test Coverage Report for Build 5092460742

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 39.621%
Totals Coverage Status
Change from base Build 5090502610: 0.0%
Covered Lines: 8876
Relevant Lines: 22402
💛 - Coveralls

bogdankostic and others added 11 commits May 24, 2023 23:46
@bogdankostic bogdankostic marked this pull request as ready for review May 26, 2023 11:57
@bogdankostic bogdankostic requested review from a team as code owners May 26, 2023 11:57
@bogdankostic bogdankostic requested review from silvanocerza and vblagoje and removed request for a team and silvanocerza May 26, 2023 11:57
vblagoje
vblagoje reviewed May 26, 2023
View reviewed changes
test/benchmarks/run.py Outdated Show resolved Hide resolved
vblagoje
vblagoje approved these changes May 26, 2023
View reviewed changes
Copy link
Member

@vblagoje vblagoje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@bogdankostic bogdankostic merged commit b8ff105 into main May 26, 2023
47 checks passed
@bogdankostic bogdankostic deleted the adapt_benchmark_run branch May 26, 2023 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vblagoje vblagoje vblagoje approved these changes

Assignees
No one assigned
Labels
topic:tests type:documentation Improvements on the docs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Adapt benchmarking script
4 participants
@bogdankostic @coveralls @vblagoje @dfokina