-
Notifications
You must be signed in to change notification settings - Fork 43.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Information Retrieval - Challenge b #4261
Conversation
Deployment failed with the following error:
|
Codecov ReportPatch and project coverage have no change.
Additional details and impacted files@@ Coverage Diff @@
## master #4261 +/- ##
=======================================
Coverage 67.77% 67.77%
=======================================
Files 72 72
Lines 3516 3516
Branches 560 560
=======================================
Hits 2383 2383
Misses 948 948
Partials 185 185 ☔ View full report in Codecov by Sentry. |
@PortlandKyGuy left some comments, here is my take : we can keep your test but precise that the information retrieval challenges also apply to using the LLM. Here is my suggestion:
This is coherent with the obtain knowledge function we plan to build: the obtain knowledge function first asks the llm and then if the llm is not confident, asks the web. |
@merwanehamadi , It looks like there were changes to the run_interaction_loop as well that were not picked up. I will work on your suggestions and fixing the run_interaction_loop for this test as well. |
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
thank you ! ❤️ |
tests/integration/challenges/information_retrieval/test_information_retrieval_challenge_b.py
Outdated
Show resolved
Hide resolved
Deployment failed with the following error:
|
@PortlandKyGuy thanks ! could you fix the linter ? don't forget to update the mkdocs.yml as well please |
@PortlandKyGuy thanks a lot! it's here now there are some things to add: |
Background
This is the information retrieval challenge b. It retrieves information that is consistent over time. It also requires enough details that a couple requests from the LLM are often needed.
Relates to Issue #3837
Changes
Added a new agent specific for this challenge (get_nobel_prize_agent) in tests/integration/agent_factory.py
Added a the new information retrieval test that is more challenging than challenge 'a', but not much more. The file is tests/integration/challenges/information_retrieval/test_information_retrieval_challenge_b.py
Documentation
The test's methods are documented and uses the same structure as challenge a.
Test Plan
This is a challenge and intended to be solved over time. Currently it is marked as skip.
PR Quality Checklist