Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Give models a retry on error #123

Closed
3 tasks done
bauersimon opened this issue May 17, 2024 · 1 comment
Closed
3 tasks done

Give models a retry on error #123

bauersimon opened this issue May 17, 2024 · 1 comment
Assignees
@ruiAzevedo19
Labels
enhancement New feature or request
Milestone
v0.5.0

Comments

@bauersimon
Copy link
Member

bauersimon commented May 17, 2024 •
edited by ruiAzevedo19
Loading

even powerful models as GPT4 and Llama3 might return EOF or an error (https://github.com/symflower/eval-dev-quality/tree/105%2B108/evaluation-2024-05-14-09%3A18%3A41), give them a second chance or something

  • add retry logic if model response is an error
  • configure retry count in the CLI
  • test this using mocks
@bauersimon bauersimon added the enhancement New feature or request label May 17, 2024
@bauersimon bauersimon added this to the v0.5.0 milestone May 17, 2024
@bauersimon bauersimon mentioned this issue May 17, 2024
Open
45 tasks
@bauersimon
Copy link
Member Author

CC @zimmski

ruiAzevedo19 added a commit that referenced this issue May 17, 2024
@ruiAzevedo19
Allow to retry a model when it errors
2c7b5de 
Part of #123
@ruiAzevedo19 ruiAzevedo19 mentioned this issue May 17, 2024
Merged
ruiAzevedo19 added a commit that referenced this issue May 17, 2024
@ruiAzevedo19
Allow to retry a model when it errors
a810a29 
Part of #123
ruiAzevedo19 added a commit that referenced this issue May 17, 2024
@ruiAzevedo19
Allow to retry a model when it errors
b5129de 
Part of #123
ruiAzevedo19 added a commit that referenced this issue May 17, 2024
@ruiAzevedo19
Allow to retry a model when it errors
3decf21 
Part of #123
ruiAzevedo19 added a commit that referenced this issue May 17, 2024
@ruiAzevedo19
Allow to retry a model when it errors
280e54e 
Part of #123
ruiAzevedo19 added a commit that referenced this issue May 17, 2024
@ruiAzevedo19
Allow to retry a model when it errors in the process of solving a task
4a67cb3 
Part of #123
Munsio pushed a commit that referenced this issue Jun 3, 2024
@ruiAzevedo19 @Munsio
Allow to retry a model when it errors in the process of solving a task
3cac63f 
Part of #123
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ruiAzevedo19 ruiAzevedo19

Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.5.0
Development

No branches or pull requests
2 participants
@bauersimon @ruiAzevedo19