Add streamGenerateContent cost tracking in passthrough #15199
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| or ("streamRawPredict") in url | ||
| ): | ||
| # Check if it's Gemini (Google AI Studio) or Vertex AI | ||
| if parsed_url.hostname and parsed_url.hostname.endswith("generativelanguage.googleapis.com"): |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 14 hours ago
The intended behavior is likely to allow requests made to generativelanguage.googleapis.com or any of its true subdomains (e.g., sub.generativelanguage.googleapis.com). We should ensure the hostname is either exactly generativelanguage.googleapis.com or ends with .generativelanguage.googleapis.com (with a dot), and not simply any string ending with that substring. This prevents hostnames like notgenerativelanguage.googleapis.com.evil.com from matching. To fix, locate the line in get_endpoint_type that does parsed_url.hostname.endswith("generativelanguage.googleapis.com") and replace it with a safe comparison ensuring it matches either exactly or a dot-subdomain. This requires no new methods but does require changing the check.
| @@ -302,7 +302,10 @@ | ||
| or ("streamRawPredict") in url | ||
| ): | ||
| # Check if it's Gemini (Google AI Studio) or Vertex AI | ||
| if parsed_url.hostname and parsed_url.hostname.endswith("generativelanguage.googleapis.com"): | ||
| if parsed_url.hostname and ( | ||
| parsed_url.hostname == "generativelanguage.googleapis.com" | ||
| or parsed_url.hostname.endswith(".generativelanguage.googleapis.com") | ||
| ): | ||
| return EndpointType.GEMINI | ||
| return EndpointType.VERTEX_AI | ||
| elif parsed_url.hostname == "api.anthropic.com": |
This reverts commit 8095de5.
Title
Add streamGenerateContent cost tracking in passthrough
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitType
🐛 Bug Fix
Changes
Problem
Passthrough streaming requests for Gemini endpoints were not calculating costs correctly. The streaming cost calculation was missing the
logging_objparameter needed for cost injection, and Gemini's unique fragmented JSON streaming format was not being parsed correctly.https://www.loom.com/share/f2b38813059b4ea3ab5259f3fae31ea5?sid=9a6b6978-553b-4d0c-a658-1a39c8c61bdb