Prompt caching does not work, cachePoint object not supported

[ShaharZivanOnvego]

To use prompt caching, according to the AWS Bedrock documentation, the system message needs to look like this:

...
  "system": [ 
    {
        "text": "You are an app that creates play lists for a radio station that plays rock and pop music. Only return song names and the artist. "
    },
    {
        "cachePoint": {
            "type": "default"
        }
    }
  ],
...

However, when trying to do this as a SystemMessage through Langchain, I get the following error:

System message must be a string, instead was: <class 'list'> Traceback (most recent call last):
  File "/home/shahar/ProjectData/AgentFramework/onvego/agent_framework/task_handler/internal.py", line 318, in __call__

If I initialize ChatBedrock with the flag beta_use_converse_api=True, then it does let me send a list - but when this list has the cachepoint object as specified:

{
    "cachePoint": {
        "type": "default",
    }
}

I get an error stating:

Parameter validation failed:
Unknown parameter in system[1]: "cachePoint", must be one of: text, guardContent

This seems to make cache prompting impossible, unless I am missing something.

[stanyq4]

As the number of our tools is constantly growing, we are also looking to utilize cachePoint param.

Is there any update on when this might be available? Sound like it should be a minor fix.

Thank you

[masquare]

Amazon Bedrock prompt caching is still in preview and as such, the new cachePoint parameter for system messages is not yet supported by botocore (see here).

We need botocore/boto3 to support the cachePoint parameter for system messages before langchain-aws can integrate it.

Tracking the boto3 release in boto/boto3#4376

[yilong016]

Amazon Bedrock prompt caching is still in preview and as such, the new cachePoint parameter for system messages is not yet supported by botocore (see here).

We need botocore/boto3 to support the cachePoint parameter for system messages before langchain-aws can integrate it.

Tracking the boto3 release in boto/boto3#4376

Dears, bedrock boto3 now supports prompt cache since boto3-1.37.24.
Can we have an update on this?
Thank you!

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/converse.html

cachePoint (dict) –

CachePoint to include in the message.

type (string) – [REQUIRED]

Specifies the type of cache point within the CachePointBlock.

[omrish-Glossai]

Looks like prompt caching was added to ChatBedrock
#369

Will it be added for ChatBedrockConverse?

[patcheng]

I was able to get prompt caching to work with ChatBedrockConverse when cachePoint is in a separate block, like:

[
    SystemMessage(
        content=[
            {
                "type": "text",
                "text": ...
            },
            {
                "cachePoint": {
                    "type": "default",
                },
            },
        ],
    ),
    HumanMessage(
        content=[
            {
                "type": "text",
                "text": ...
            },
            {
                "cachePoint": {
                    "type": "default",
                },
            },
        ],
    ),
]

[wcvb13]

prompt = ChatPromptTemplate.from_messages( [ SystemMessage( content=[ { "type": "text", "text": "....." } ], ), HumanMessage( content=[ { "type": "placeholder", "text": "{messages}" }, { "cachePoint": { "type": "default", }, }, ] ), ] ).partial(time=datetime.now)

I checked the cache_write_input_tokens in the metadata from the test code above, but it turned out to be empty.

[omrish-Glossai]

I was able to get prompt caching to work with ChatBedrockConverse when cachePoint is in a separate block, like:

[
    SystemMessage(
        content=[
            {
                "type": "text",
                "text": ...
            },
            {
                "cachePoint": {
                    "type": "default",
                },
            },
        ],
    ),
    HumanMessage(
        content=[
            {
                "type": "text",
                "text": ...
            },
            {
                "cachePoint": {
                    "type": "default",
                },
            },
        ],
    ),
]

This is great! Worked for me as well.
Using tools raises issues though.
I got this one when trying to add the cachePoint dict into my tools after parsing them.

File "...\.venv\Lib\site-packages\langchain_core\utils\function_calling.py", line 490, in convert_to_openai_function raise ValueError(msg) ValueError: Unsupported function {'cachePoint': {'type': 'default'}} Functions must be passed in as Dict, pydantic.BaseModel, or Callable. If they're a dict they must either be in OpenAI function format or valid JSON schema with top-level 'title' and 'description' keys.

[sbogdan]

I was able to get prompt caching to work with ChatBedrockConverse when cachePoint is in a separate block, like:

[
    SystemMessage(
        content=[
            {
                "type": "text",
                "text": ...
            },
            {
                "cachePoint": {
                    "type": "default",
                },
            },
        ],
    ),
    HumanMessage(
        content=[
            {
                "type": "text",
                "text": ...
            },
            {
                "cachePoint": {
                    "type": "default",
                },
            },
        ],
    ),
]

This is great! Worked for me as well. Using tools raises issues though. I got this one when trying to add the cachePoint dict into my tools after parsing them.

File "...\.venv\Lib\site-packages\langchain_core\utils\function_calling.py", line 490, in convert_to_openai_function raise ValueError(msg) ValueError: Unsupported function {'cachePoint': {'type': 'default'}} Functions must be passed in as Dict, pydantic.BaseModel, or Callable. If they're a dict they must either be in OpenAI function format or valid JSON schema with top-level 'title' and 'description' keys.

omrish-Glossai - have got it working, but not something I'd wholeheartedly recommend: you can pass a "patched" boto client to ChatBedrockConverse, and have the overridden converse method add a cache point where needed (according to aws docs -> tools checkpoints). Obviously fragile, but might help until support is properly added.

class PatchedBedrockConverseClient:
    def __init__(self):
        self._client = boto3.client('bedrock-runtime')

    def converse(self, **kwargs):
        tools = kwargs.get('toolConfig', {}).get('tools')
        if tools:
            tools.append({"cachePoint": {"type": "default"}})
        return self._client.converse(**kwargs)

ChatBedrockConverse(
    model="<model_id>",
    # temperature, etc.
    client=PatchedBedrockConverseClient()
).with_structured_output(SomeOutputSchemaType) # or whatever other tools are defined

[michaelnchin]

Support for prompt caching with tools has been added to ChatBedrockConverse with #460, and is available in the v0.2.22 release.