Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add notebook for RAG eval harness #11

Merged
merged 1 commit into from
Jul 8, 2024

Conversation

shadeMe
Copy link
Contributor

@shadeMe shadeMe commented Jun 6, 2024

Proposed Changes:

Add a notebook to showcase the RAG evaluation harness.

Notes for the reviewer

Depends on the following: deepset-ai/haystack#7818

Without the above PR, the execution of the harness will break since it attempts to serialize the evaluation pipeline. To test it locally, either pull the above PR and build haystack-ai locally or remove the answer faithfulness metric from the harness.

Checklist

@shadeMe shadeMe requested a review from a team as a code owner June 6, 2024 13:20
@shadeMe shadeMe requested review from julian-risch, davidsbatista, bilgeyucel and a team and removed request for a team and julian-risch June 6, 2024 13:20
@coveralls
Copy link

coveralls commented Jun 6, 2024

Pull Request Test Coverage Report for Build 9401655773

Details

  • 20 of 22 (90.91%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 97.764%

Changes Missing Coverage Covered Lines Changed/Added Lines %
haystack_experimental/evaluation/harness/rag/harness.py 20 22 90.91%
Totals Coverage Status
Change from base Build 9352626696: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@TuanaCelik
Copy link
Member

@davidsbatista
Copy link
Contributor

I think the code is already there, and it shows how to use it. Nevertheless, this notebook needs some improvement regarding "documentation" and the code's organisation. Here are my comments:

@davidsbatista
Copy link
Contributor

just noticed that the keyword_eval_harness is not used anywhere

@davidsbatista
Copy link
Contributor

@davidsbatista
Copy link
Contributor

@shadeMe can you open another PR only with the pytoml ?

@shadeMe shadeMe force-pushed the feat/rag-eval-harness-notebook branch from 29d2bfc to 7b48f64 Compare June 13, 2024 10:57
@coveralls
Copy link

coveralls commented Jun 13, 2024

Pull Request Test Coverage Report for Build 9498393193

Details

  • 20 of 22 (90.91%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 97.764%

Changes Missing Coverage Covered Lines Changed/Added Lines %
haystack_experimental/evaluation/harness/rag/harness.py 20 22 90.91%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@shadeMe shadeMe force-pushed the feat/rag-eval-harness-notebook branch from 7b48f64 to abd14f5 Compare July 5, 2024 11:21
@shadeMe shadeMe marked this pull request as draft July 5, 2024 11:21
@shadeMe shadeMe force-pushed the feat/rag-eval-harness-notebook branch from abd14f5 to 81f1bba Compare July 5, 2024 13:27
@shadeMe shadeMe force-pushed the feat/rag-eval-harness-notebook branch from 81f1bba to 8a2a3eb Compare July 8, 2024 11:17
@shadeMe shadeMe marked this pull request as ready for review July 8, 2024 11:17
@shadeMe shadeMe merged commit 7592cc5 into deepset-ai:main Jul 8, 2024
@shadeMe shadeMe deleted the feat/rag-eval-harness-notebook branch July 8, 2024 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants