[Bug]: Caching completely broken with cache_control parameter when using PromptCachingDeploymentCheck

[hnykda]

What happened?

1. We use LiteLLM's Proxy PromptCachingDeploymentCheck to route requests from the same conversation to the same provider, with the goal of increasing cache hits.
2. That routing step checks the length of the messages since anything under 1024 tokens won't be cached.
3. However, LiteLLM cannot count the length of cacheable messages. They cause a ValueError, which is interpreted as being uncacheable.

An error we get:

  "message": "Error in is_prompt_caching_valid_prompt: Unsupported type <class 'dict'> for key cache_control in message {'content': 'You are presented with the following task:

...our truncated message...

tools.\\n', 'role': 'user', 'cache_control': {'type': 'ephemeral'}}",

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.72.2.rc

Twitter / LinkedIn details

No response

[bml1g12]

I have encountered same issue, seems it was stable on litellm==1.66.2 but as of latest 1.74.0 this is buggy

Code excerpt:

       cached_user_message = {
            # Bedrock requires at least one non-system message
            "role": "user",
            # https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
            # 5 min TTL cache
            "content": [
                {
                    "type": "text",
                    "text": f"{statement1}\n{statement2}",
                    "cache_control": {"type": "ephemeral"},
                }
            ],
        }

        self._litellm_params = LiteLLMCompletionParams(
            model=self._model["id"],
            tool_choice=None,
            response_format=AssessmentOutput,
            temperature=0,
            aws_region_name=self._model["aws_region_name"],
            messages=[system_message, cached_user_message],
        )