Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Information Retrieval - Challenge b #4261

Closed
wants to merge 19 commits into from

Conversation

PortlandKyGuy
Copy link
Contributor

@PortlandKyGuy PortlandKyGuy commented May 17, 2023

Background

This is the information retrieval challenge b. It retrieves information that is consistent over time. It also requires enough details that a couple requests from the LLM are often needed.

Relates to Issue #3837

Changes

Added a new agent specific for this challenge (get_nobel_prize_agent) in tests/integration/agent_factory.py

Added a the new information retrieval test that is more challenging than challenge 'a', but not much more. The file is tests/integration/challenges/information_retrieval/test_information_retrieval_challenge_b.py

Documentation

The test's methods are documented and uses the same structure as challenge a.

Test Plan

This is a challenge and intended to be solved over time. Currently it is marked as skip.

PR Quality Checklist

  • My pull request is atomic and focuses on a single change.
  • I have thoroughly tested my changes with multiple different prompts.
  • I have considered potential risks and mitigations for my changes.
  • I have documented my changes clearly and comprehensively.
  • I have not snuck in any "extra" small tweaks changes

@vercel
Copy link

vercel bot commented May 17, 2023

Deployment failed with the following error:

Resource is limited - try again in 2 hours (more than 100, code: "api-deployments-free-per-day").

@codecov
Copy link

codecov bot commented May 17, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (ee9f10a) 67.77% compared to head (d8811a4) 67.77%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4261   +/-   ##
=======================================
  Coverage   67.77%   67.77%           
=======================================
  Files          72       72           
  Lines        3516     3516           
  Branches      560      560           
=======================================
  Hits         2383     2383           
  Misses        948      948           
  Partials      185      185           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@waynehamadi
Copy link
Contributor

@PortlandKyGuy left some comments, here is my take : we can keep your test but precise that the information retrieval challenges also apply to using the LLM.

Here is my suggestion:

  • add the web command to the challenge
  • change CYCLE_COUNT to 2
  • normally this challenge should be done in 2 cycles. (1 to write to file and 1 to task_complete) because it has to use the LLM to get the answer.

This is coherent with the obtain knowledge function we plan to build: the obtain knowledge function first asks the llm and then if the llm is not confident, asks the web.

@PortlandKyGuy
Copy link
Contributor Author

@merwanehamadi , It looks like there were changes to the run_interaction_loop as well that were not picked up. I will work on your suggestions and fixing the run_interaction_loop for this test as well.

@vercel
Copy link

vercel bot commented May 17, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 29, 2023 0:22am

@waynehamadi
Copy link
Contributor

thank you ! ❤️
@PortlandKyGuy

@vercel vercel bot temporarily deployed to Preview May 20, 2023 20:42 Inactive
@vercel vercel bot temporarily deployed to Preview May 26, 2023 13:33 Inactive
@vercel
Copy link

vercel bot commented May 26, 2023

Deployment failed with the following error:

Resource is limited - try again in 10 minutes (more than 100, code: "api-deployments-free-per-day").

@waynehamadi
Copy link
Contributor

@PortlandKyGuy thanks ! could you fix the linter ?
Also we need this in the doc, you can copy how it's been done with docs/challenges/information_retrieval/challenge_a.md

don't forget to update the mkdocs.yml as well please

@vercel vercel bot temporarily deployed to Preview May 28, 2023 04:27 Inactive
@vercel vercel bot temporarily deployed to Preview May 29, 2023 00:22 Inactive
@waynehamadi waynehamadi mentioned this pull request May 29, 2023
6 tasks
@waynehamadi
Copy link
Contributor

@PortlandKyGuy thanks a lot! it's here now there are some things to add:
#4456

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

3 participants