You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like to suggest adding an alternative to the default routing strategy. As I understand it, requests are currently routed sequentially using a round-robin approach. It would be very useful to have an additional mode where the API key is selected randomly from the available list.
This would be especially helpful when working with a large number of keys, as it could distribute token usage more evenly across them. In my case, under heavy usage, the first 10 out of 100 API keys were blocked fairly quickly, even though the total key pool is much larger.
It would be great if CLI-PROXY-API supported a random strategy in the routing configuration, for example:
config.yaml:
routing:
strategy: "round-robin"# round-robin (default), fill-first, random
Thank you for considering this feature request. I believe it would be very useful for users managing a large set of API keys.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
I would like to suggest adding an alternative to the default routing strategy. As I understand it, requests are currently routed sequentially using a
round-robinapproach. It would be very useful to have an additional mode where the API key is selected randomly from the available list.This would be especially helpful when working with a large number of keys, as it could distribute token usage more evenly across them. In my case, under heavy usage, the first 10 out of 100 API keys were blocked fairly quickly, even though the total key pool is much larger.
It would be great if
CLI-PROXY-APIsupported arandomstrategy in theroutingconfiguration, for example:config.yaml:
Thank you for considering this feature request. I believe it would be very useful for users managing a large set of API keys.
Beta Was this translation helpful? Give feedback.
All reactions