Information Retrieval - Challenge b

[PortlandKyGuy]

Background

This is the information retrieval challenge b. It retrieves information that is consistent over time. It also requires enough details that a couple requests from the LLM are often needed.

Relates to Issue #3837

Changes

Added a new agent specific for this challenge (get_nobel_prize_agent) in tests/integration/agent_factory.py

Added a the new information retrieval test that is more challenging than challenge 'a', but not much more. The file is tests/integration/challenges/information_retrieval/test_information_retrieval_challenge_b.py

Documentation

The test's methods are documented and uses the same structure as challenge a.

Test Plan

This is a challenge and intended to be solved over time. Currently it is marked as skip.

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 2 hours (more than 100, code: "api-deployments-free-per-day").

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (ee9f10a) 67.77% compared to head (d8811a4) 67.77%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4261   +/-   ##
=======================================
  Coverage   67.77%   67.77%           
=======================================
  Files          72       72           
  Lines        3516     3516           
  Branches      560      560           
=======================================
  Hits         2383     2383           
  Misses        948      948           
  Partials      185      185           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[waynehamadi]

@PortlandKyGuy left some comments, here is my take : we can keep your test but precise that the information retrieval challenges also apply to using the LLM.

Here is my suggestion:

• add the web command to the challenge
• change CYCLE_COUNT to 2
• normally this challenge should be done in 2 cycles. (1 to write to file and 1 to task_complete) because it has to use the LLM to get the answer.

This is coherent with the obtain knowledge function we plan to build: the obtain knowledge function first asks the llm and then if the llm is not confident, asks the web.

[PortlandKyGuy]

@merwanehamadi , It looks like there were changes to the run_interaction_loop as well that were not picked up. I will work on your suggestions and fixing the run_interaction_loop for this test as well.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 29, 2023 0:22am

[waynehamadi]

thank you ! ❤️
@PortlandKyGuy

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 10 minutes (more than 100, code: "api-deployments-free-per-day").

[waynehamadi]

@PortlandKyGuy thanks ! could you fix the linter ?
Also we need this in the doc, you can copy how it's been done with docs/challenges/information_retrieval/challenge_a.md

don't forget to update the mkdocs.yml as well please

[waynehamadi]

@PortlandKyGuy thanks a lot! it's here now there are some things to add:
#4456