Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rate limit across batches, not within batches #235

Open
ianarawjo opened this issue Mar 5, 2024 · 1 comment
Open

Rate limit across batches, not within batches #235

ianarawjo opened this issue Mar 5, 2024 · 1 comment
Labels
low priority Would be nice to have, but not mission-critical

Comments

@ianarawjo
Copy link
Owner

ChainForge currently does rate-limiting but within batches of requests to model providers. If one calls queryLLM lots of times for single calls, the rate limiting won't be enforced. To fix this, we should rate-limit at the level of the API request itself ---i.e., in call_chatgpt or call_anthropic, using a rate-limiting library that works on requests per minute (on time).

Rate limits should also be updated to reflect current standards.

@ianarawjo ianarawjo added the low priority Would be nice to have, but not mission-critical label Mar 5, 2024
@ianarawjo
Copy link
Owner Author

This change is also complete but on the Typescript frontend branch. It will be pushed in that update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
low priority Would be nice to have, but not mission-critical
Projects
None yet
Development

No branches or pull requests

1 participant