Record cached tokens

[hadley]

Since caching is automatic, and it might actually might a difference to the cost.

[christophscheuch]

It should make a difference to cost since they quote dedicated prices for cached input tokens: https://openai.com/index/api-prompt-caching/.

What makes a more general solution for this particularly complicated is the fact that Anthropic has a pricing schedule that differentiates between 5-minute and 1-hour cache writes: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching.

[hadley]

@christophscheuch yes, that's why this issue is specifically about chatGPT.

[hadley]

And gemini now does automatic completion: https://ai.google.dev/gemini-api/docs/caching?lang=python