Problem
The onboarding picker's "Another supported vendor (type to search)" step offers the
117 vendors LiteLLM routes to. Every one of them that Raven carries no spec for is
treated as needing a single API key: the wizard prompts for one, stores it, and
reports success. For at least nine of those vendors that produces a provider that
cannot make a call.
Measured against litellm 1.85.0:
Authenticate by device flow, and take no API key at all
| Vendor |
Evidence |
chatgpt |
ships llms/chatgpt/authenticator.py |
gigachat |
ships llms/gigachat/authenticator.py |
Raven registers provider login for four families only (openai_codex,
github_copilot, minimax_global, minimax_cn), so a user who configures either
of these can never obtain a token: the key they typed is never read, and the first
call starts a browser device flow.
Need more than one credential, or a credential that is not a key
| Vendor |
validate_environment reports |
bedrock |
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY |
sagemaker |
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY |
vertex_ai |
VERTEXAI_PROJECT, VERTEXAI_LOCATION |
azure |
AZURE_API_BASE, AZURE_API_KEY, AZURE_API_VERSION |
cloudflare |
CLOUDFLARE_API_BASE, CLOUDFLARE_API_KEY |
datarobot |
DATAROBOT_API_TOKEN |
This is not a LiteLLM shortcoming. AWS signs requests with an access key, a secret
and a region; GCP uses a service account plus a project and a location; Azure gives
every tenant its own resource URL. Those are the vendors' own auth models, and
LiteLLM reflects them faithfully. The wrong part is Raven's assumption that a
vendor with no spec of ours needs exactly one API key.
A further 85 of the 117 could not be classified by validate_environment at all.
That is a limit of the probe, not evidence that they are fine.
Bedrock and Vertex AI are the ones likely to be missed: neither has a spec, so the
vendor step is the only way to reach them, and it cannot collect their credentials.
Scope
Not a regression of any one change: opening the picker up to every supported vendor
made an existing assumption reachable. Azure OpenAI had the same shape while being
in the curated list, and is fixed separately.
Possible directions
- Collect what the vendor actually needs.
litellm.validate_environment(model=...)
returns the missing variables per provider, correctly for bedrock, vertex_ai and
azure among others. The wizard would prompt for those fields rather than assuming
one key. ProviderConfig currently holds api_key, api_base, extra_headers
and models, so it would need somewhere to keep the rest.
- Offer only what can be configured. Filter the vendor list to the shapes the
wizard can actually collect, and give the others a spec (with a login flow where
the vendor uses one) when someone wants them. Smaller, and it stops promising
something that does not work.
Worth noting for direction 1: validate_environment itself resolves credentials,
so calling it for github_copilot starts a GitHub device login and blocks for
minutes. Route any such call through _may_prompt in
raven/token_wise/pricing.py, which answers "would handing this model to LiteLLM
start an interactive login" by inspecting the installed package.
Reproduce
raven onboard, pick "Another supported vendor", type bedrock.
- The wizard asks for one API key and reports the provider configured.
- Nothing supplies
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, so no call can be
signed.
Problem
The onboarding picker's "Another supported vendor (type to search)" step offers the
117 vendors LiteLLM routes to. Every one of them that Raven carries no spec for is
treated as needing a single API key: the wizard prompts for one, stores it, and
reports success. For at least nine of those vendors that produces a provider that
cannot make a call.
Measured against litellm 1.85.0:
Authenticate by device flow, and take no API key at all
chatgptllms/chatgpt/authenticator.pygigachatllms/gigachat/authenticator.pyRaven registers
provider loginfor four families only (openai_codex,github_copilot,minimax_global,minimax_cn), so a user who configures eitherof these can never obtain a token: the key they typed is never read, and the first
call starts a browser device flow.
Need more than one credential, or a credential that is not a key
validate_environmentreportsbedrockAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYsagemakerAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYvertex_aiVERTEXAI_PROJECT,VERTEXAI_LOCATIONazureAZURE_API_BASE,AZURE_API_KEY,AZURE_API_VERSIONcloudflareCLOUDFLARE_API_BASE,CLOUDFLARE_API_KEYdatarobotDATAROBOT_API_TOKENThis is not a LiteLLM shortcoming. AWS signs requests with an access key, a secret
and a region; GCP uses a service account plus a project and a location; Azure gives
every tenant its own resource URL. Those are the vendors' own auth models, and
LiteLLM reflects them faithfully. The wrong part is Raven's assumption that a
vendor with no spec of ours needs exactly one API key.
A further 85 of the 117 could not be classified by
validate_environmentat all.That is a limit of the probe, not evidence that they are fine.
Bedrock and Vertex AI are the ones likely to be missed: neither has a spec, so the
vendor step is the only way to reach them, and it cannot collect their credentials.
Scope
Not a regression of any one change: opening the picker up to every supported vendor
made an existing assumption reachable. Azure OpenAI had the same shape while being
in the curated list, and is fixed separately.
Possible directions
litellm.validate_environment(model=...)returns the missing variables per provider, correctly for bedrock, vertex_ai and
azure among others. The wizard would prompt for those fields rather than assuming
one key.
ProviderConfigcurrently holdsapi_key,api_base,extra_headersand
models, so it would need somewhere to keep the rest.wizard can actually collect, and give the others a spec (with a login flow where
the vendor uses one) when someone wants them. Smaller, and it stops promising
something that does not work.
Worth noting for direction 1:
validate_environmentitself resolves credentials,so calling it for
github_copilotstarts a GitHub device login and blocks forminutes. Route any such call through
_may_promptinraven/token_wise/pricing.py, which answers "would handing this model to LiteLLMstart an interactive login" by inspecting the installed package.
Reproduce
raven onboard, pick "Another supported vendor", typebedrock.AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, so no call can besigned.