Manual Caching (Anthropic)

[hewliyang]

chatlas/chatlas/_provider_anthropic.py

Lines 614 to 616 in 0ac18d2

	# N.B. Currently, Anthropic doesn't cache by default and we currently do not support
	# manual caching in chatlas. Note also that this only tracks reads, NOT writes, which
	# have their own cost. To track that properly, we would need another caching category and per-token cost.

Many-turn agentic use cases are simply not feasible cost wise without caching.

Are there any plans to bring in support for manual caching?

I think since Turn's support the system role, a relatively straightfoward and non-invasive way of doing this would be to expose a callback that will run before transforming list[Turn] to list[ProviderXYZ's Messages]

i.e.

def cache_last_message(turns: List[Turn]) -> List[Turn]:
    if not turns: return turns
    
    for content in turns[-1].contents:
        content[-1].cache_control = {"type": "ephemeral"}

    return turns

chat = ChatAnthropic(...)
chat.set_turn_callback(cache_system_and_last)

[cpsievert]

This is a great point, thanks for filing. This problem is fairly high up the priority list, but may not get to it for a few weeks or so. I do, however, think I like the proposal -- would you like to put a PR together?

[cpsievert]

#215 adds in caching support by default -- please let me know if it satisfies your needs, thanks!