Support for Bedrock Prompt Caching in LLM-as-a-Judge

[saefty]

Describe the feature or potential improvement

In our workload many answers are identical throughout specific requests patterns and time of day resulting in unessecary LLM-as-a-Judge evaluations.

We see Bedrock Prompt Caching as an ideal solution to reduce LLM as a judge costs while retaining meaningful score metric datapoints and reducing pipeline times.

References:

• langfuse/packages/shared/src/server/llm/fetchLLMCompletion.ts

  Line 241 in a3b407a

  chatModel = new ChatBedrockConverse({

• https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
• Prompt caching does not work, cachePoint object not supported langchain-ai/langchain-aws#326 (comment)

Additional information

No response