Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Bugs & Improve Semi Automated Quality Evaluation DAG #303

Merged
merged 5 commits into from
Feb 23, 2024

Conversation

davidgxue
Copy link
Collaborator

@davidgxue davidgxue commented Feb 21, 2024

Description

  • The subset of questions parameter is not working previously for the DAG. It was not able to parse a list of question numbers taken in as input and causes errors (outlined in the issue linked below)
  • weaviate_search_multiquery_retriever is deleted: it is no longer relevant or useful. It was not a good practice to hardcode a way of doing multiquery retrieval in the evaluation run DAG since the retrieval chain in the api server can change anytime. This would not produce related result in such a scenario and do not add additional data insights. (E.g. right now we do multiquery differently, uses hybrid search and reranker, none of these are taken into account)
  • weaviate_search_references is deleted. Similar reason as above. The actual RAG pipeline now uses hybrid search + reranker. There's no reason to query for results from weaviate using an embedding model. It is not a comparable result.
  • Renamed DAG and files for better readability

Tests

image

closes #298

Copy link

cloudflare-pages bot commented Feb 21, 2024

Deploying with  Cloudflare Pages  Cloudflare Pages

Latest commit: f88ba8e
Status: ✅  Deploy successful!
Preview URL: https://e1c86666.ask-astro.pages.dev
Branch Preview URL: https://fix-automated-testing-dag.ask-astro.pages.dev

View logs

@davidgxue davidgxue self-assigned this Feb 22, 2024
@davidgxue davidgxue marked this pull request as ready for review February 22, 2024 08:43
@davidgxue davidgxue changed the title [WIP/Draft] [Not Ready for Review] Fix Bugs & Improve Semi Automated Evaluation DAG Fix Bugs & Improve Semi Automated Evaluation DAG Feb 22, 2024
@davidgxue davidgxue changed the title Fix Bugs & Improve Semi Automated Evaluation DAG Fix Bugs & Improve Semi Automated Quality Evaluation DAG Feb 23, 2024
Copy link
Collaborator

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@davidgxue davidgxue merged commit e180610 into main Feb 23, 2024
8 checks passed
@davidgxue davidgxue deleted the fix-automated-testing-dag branch February 23, 2024 04:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve & Address Bugs intest_retrieval the Batch Test Question DAG
2 participants