refactor: Add reader-retriever benchmark script

[bogdankostic]

Related Issues

• related to Adapt benchmarking script #4902

Related PRs

• depends on refactor: Generate eval result in separate method #5001
• depends on refactor: Adapt benchmarking utils #5003
• depends on refactor: Adapt retriever benchmarks script #5004
• depends on refactor: Adapt reader benchmarks #5005

Proposed Changes:

This PR adds a file benchmarks/retriever_reader.py that allows to run benchmarks on a retriever-reader Pipeline by providing an indexing pipeline, a querying pipeline, and an evaluation set.

How did you test it?

Manual tests.

Notes for the reviewer

We use run_batch instead of eval_batch methods as measuring the time of eval isn’t representative for pure inference speed because it creates a significant overhead.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 5089467124

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 39.621%

---

Totals
Change from base Build 5089433768:	0.0%
Covered Lines:	8876
Relevant Lines:	22402

---

💛 - Coveralls

[rjanjua]

When trying to run this benchmarl I get the following error:

Traceback (most recent call last):
  File "/Users/rohan/repos/deepset/haystack/test/benchmarks/run.py", line 8, in <module>
    from reader import benchmark_reader
  File "/Users/rohan/repos/deepset/haystack/test/benchmarks/reader.py", line 1, in <module>
    from utils import get_document_store, index_to_doc_store, get_reader
ImportError: cannot import name 'get_document_store' from 'utils' (/Users/rohan/repos/deepset/haystack/test/benchmarks/utils.py)

[bogdankostic]

@rjanjua I merged #5004 and #5005 into this PR's branch, can you try again?

[rjanjua]

@bogdankostic I have a new error for you now 😄

  File "/Users/rohan/repos/deepset/haystack/test/benchmarks/run.py", line 9, in <module>
    from utils import load_config
ImportError: cannot import name 'load_config' from 'utils' (/Users/rohan/repos/deepset/haystack/test/benchmarks/utils.py)