Self Hosted GPU: Human Priority, then agents when idle.

[Noriller]

Not sure if https://docs.litellm.ai/docs/scheduler would cover this use case, but:

I want humans to use the model without sharing with bots/agents. If the model is idle for enough time, then let bot requests through.

Agents can handle and backoff of 429 errors. So let them use the GPU if no one is using.

Since the GPU it can handle some parallel requests, then only simple priority might not be enough and I don't want to limit parallel requests on idle time.

Once a request started, not a problem to let it finish. But don't want to compete slots with something that can run on background when idle.