Add bedrock passthrough logging handler for cost tracking etc #11569
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
|
There was a problem hiding this comment.
Pull Request Overview
Adds support for logging and cost tracking of Amazon Bedrock passthrough requests, integrates a new Bedrock handler, and updates routing and request plumbing.
- Introduce
BedrockPassthroughLoggingHandlerto transform and log Bedrock responses - Extend pass-through routing (
pass_through_request,llm_passthrough_endpoints,success_handler) to carrymodelinfo and invoke the Bedrock handler - Add comprehensive unit tests for Anthropic and Titan response formats under Bedrock
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/pass_through_unit_tests/test_bedrock_passthrough_logging.py | New tests covering Anthropic, Titan, generic, cost, error and metadata behaviors |
| litellm/proxy/pass_through_endpoints/success_handler.py | Import and wire up Bedrock handler and route detection logic |
| litellm/proxy/pass_through_endpoints/pass_through_endpoints.py | Add model param to pass-through request signature and propagate to handlers |
| litellm/proxy/pass_through_endpoints/llm_provider_handlers/bedrock_passthrough_logging_handler.py | Implement Bedrock response transformation & logging payload builder |
| litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py | Extract model from Bedrock request body and pass into the pass-through plumbing |
Comments suppressed due to low confidence (3)
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py:487
- The
modelparameter was added topass_through_requestbut the function docstring and parameter description were not updated. Please document the new parameter to keep the signature and docs in sync.
model: Optional[str] = None,
tests/pass_through_unit_tests/test_bedrock_passthrough_logging.py:236
- [nitpick] The generic Bedrock response test does not assert the
finish_reasonor usage fields; adding those assertions would ensure consistent behavior across all formats.
assert model_response.choices[0].message.content == "This is a generic response from Titan."
litellm/proxy/pass_through_endpoints/success_handler.py:47
- The
is_bedrock_routelogic checks for substrings 'bedrock-runtime' inurl_route, buturl_routepassed in is the path (e.g., '/model/.../invoke') so it never matches. Update the tracked patterns or matching logic to use the correct path segment.
self.TRACKED_BEDROCK_ROUTES = [
...lm/proxy/pass_through_endpoints/llm_provider_handlers/bedrock_passthrough_logging_handler.py
Show resolved
Hide resolved
| return metadata.get("user") | ||
|
|
||
| @staticmethod | ||
| def _should_log_request(kwargs: dict) -> bool: |
tests/pass_through_unit_tests/test_bedrock_passthrough_logging.py
Outdated
Show resolved
Hide resolved
| model = kwargs.get("model", "unknown") | ||
|
|
||
| # Get the appropriate Bedrock configuration | ||
| bedrock_config = ProviderConfigManager.get_provider_chat_config( |
There was a problem hiding this comment.
how will we know if it's a converse or invoke transformation?
There was a problem hiding this comment.
i assume you need to check the url route and set this on the model - invoke/<model> or converse/<model>
| @@ -40,6 +43,12 @@ def __init__(self): | |||
| # Anthropic | |||
| self.TRACKED_ANTHROPIC_ROUTES = ["/messages"] | |||
|
|
|||
| # Bedrock | |||
| self.TRACKED_BEDROCK_ROUTES = [ | |||
| "bedrock-runtime", | |||
There was a problem hiding this comment.
we should be more precise here and say invoke and converse are supported - as the transformation in the bedrock_passthrough_logging_handler is configured for that.
|
Closing as cost tracking via passthrough for bedrock is supported |
Title
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitType
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes