[Feature]: Support Modal Proxy Authentication #11021
Description
The Feature
Using the proxy I have written a custom handler to set-up the Modal credentials and URL etc, which I have configured and it is being called, however, the proxy is then calling Modal without passing the proxy auth credentials. The example curl that work for me is:
curl -X POST
-H "Content-Type: application/json"
-H "Modal-Key: $TOKEN_ID"
-H "Modal-Secret: $TOKEN_SECRET"
-d '{"prompt": "Why is the sky blue?"}'
https://--meta-llama-3-8b-instruct-web.modal.run/
When I run the curl agains the proxy:
curl -X POST "http://localhost:4000/v1/chat/completions"
-H "Content-Type: application/json"
-H "Authorization: Bearer sk-wW3bVkTN054w0iORgIrPNQ"
-d '{
"model": "custom/llama3-modal",
"messages": [
{
"role": "user",
"content": "Count from 1 to 10 slowly"
}
]
}'
{"error":{"message":"litellm.APIConnectionError: CustomException - modal-http: missing credentials for proxy authorization\n. Received Model Group=custom/llama3-modal\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}%
Motivation, pitch
I want to be able use use LLMs on Modal with the containers protected by Modal's Proxy Auth capability to prevent containers with GPU(s) being spun up with associated costs for invalid requests
Are you a ML Ops Team?
No
Twitter / LinkedIn details
No response