The Feature
When Anthropic-style explicit prompts caching is used, LiteLLM currently returns (and also logs to SpendLogs) the usage object that contains the "cache_creation_input_tokens" attribute e.g.
"usage": {
"completion_tokens": 467,
"prompt_tokens": 31,
"total_tokens": 498,
"prompt_tokens_details": {
"cached_tokens": 0
},
"cache_creation_input_tokens": 1253,
"cache_read_input_tokens": 0
}
Even if 1hr cache control was used in the request "cache_control": {"type": "ephemeral", "ttl": "1h"}
Internally, LiteLLM already distinguishes between 5h vs 1h ephemeral cache for costs calculation, the codebase is searchable by cache_creation_token_details object.
The ask is to begin returning this object (with 5m vs 1hr creation details) in usage response and also log it to Postgre database Litellm_SpendLogs "metadata" json
Motivation, pitch
-
For usage scenarios where cost calculations are sensitive and important, a user may not be able to rely solely on just-in-time cost calculations by LiteLLM itself, given a possibility that there was a pricing mistake in models info, bugs in LiteLLM cost calculation logic, and any other situations where a user may want to independently recalculate and revalidate requests costs. Most of the input information is already available in LiteLLM_SpendLogs table, but not the 5min vs 1hr cache creation breakdown.
-
For corner case usage scenarios where there is a chain of LiteLLMs (e.g. master organization-level LiteLLM + second level departmental LiteLLM), it would be beneficial that the first LiteLLM has piped enough usage information into the "usage" object that the downstream LiteLLM (or another different gateway proxy) could make its own pricing calculations. Currently, the information is lost and 5min vs 1hr cache creation breakdown is not provided to the client.
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
No
Twitter / LinkedIn details
No response
The Feature
When Anthropic-style explicit prompts caching is used, LiteLLM currently returns (and also logs to SpendLogs) the usage object that contains the "cache_creation_input_tokens" attribute e.g.
Even if 1hr cache control was used in the request "cache_control": {"type": "ephemeral", "ttl": "1h"}
Internally, LiteLLM already distinguishes between 5h vs 1h ephemeral cache for costs calculation, the codebase is searchable by
cache_creation_token_detailsobject.The ask is to begin returning this object (with 5m vs 1hr creation details) in usage response and also log it to Postgre database Litellm_SpendLogs "metadata" json
Motivation, pitch
For usage scenarios where cost calculations are sensitive and important, a user may not be able to rely solely on just-in-time cost calculations by LiteLLM itself, given a possibility that there was a pricing mistake in models info, bugs in LiteLLM cost calculation logic, and any other situations where a user may want to independently recalculate and revalidate requests costs. Most of the input information is already available in LiteLLM_SpendLogs table, but not the 5min vs 1hr cache creation breakdown.
For corner case usage scenarios where there is a chain of LiteLLMs (e.g. master organization-level LiteLLM + second level departmental LiteLLM), it would be beneficial that the first LiteLLM has piped enough usage information into the "usage" object that the downstream LiteLLM (or another different gateway proxy) could make its own pricing calculations. Currently, the information is lost and 5min vs 1hr cache creation breakdown is not provided to the client.
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
No
Twitter / LinkedIn details
No response