Support for Prompt Caching?

[kharbat]

Hello, thanks for the nice work. I'm looking to replace my from-scratch architecture and give this a try - I was wondering if it supports prompt caching

[ilvalerione]

Hi @kharbat, what do you mean with prompt caching? What's the use case you are thinking of?

[kharbat]

Basically to tell the AI that certain prompts contain repetative content/instructions.

https://platform.openai.com/docs/guides/prompt-caching

[ilvalerione]

I'll take a look on that. It's a similar discussion as #285 maybe we can come up with a good and consistent design together.