Commit da9acc6
authored
🤖 fix: enable prompt caching for mux-gateway Anthropic models (#816)
## Problem
Prompt caching wasn't working for Anthropic models accessed via
mux-gateway. Only the system message (~5.4k tokens) was being cached,
but conversation history was not.
## Root Cause
The gateway provider uses a different request format and translation
mechanism:
| Provider | Request Format | Cache Control Translation |
|----------|---------------|--------------------------|
| Direct Anthropic | `json.messages` | SDK translates `providerOptions`
at all levels |
| Gateway | `json.prompt` | Gateway server only translates
`providerOptions` at **message level** |
Our `applyCacheControl()` was setting
`providerOptions.anthropic.cacheControl` at the **content part level**,
which the gateway server ignores. Only `createCachedSystemMessage()` was
setting it at message level (hence system prompt caching worked).
## Fix
Update `wrapFetchWithAnthropicCacheControl` to:
1. Detect gateway format by checking for `json.prompt` array
2. Add `providerOptions.anthropic.cacheControl` at **message level** for
gateway requests
3. Keep `cache_control` injection at content part level for direct
Anthropic
## Testing
Verified caching works by checking that cache read tokens increase on
subsequent messages.
---
_Generated with `mux`_1 parent 22a816f commit da9acc6
1 file changed
+23
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
125 | 125 | | |
126 | 126 | | |
127 | 127 | | |
128 | | - | |
129 | | - | |
130 | | - | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
131 | 149 | | |
| 150 | + | |
| 151 | + | |
132 | 152 | | |
133 | | - | |
134 | 153 | | |
135 | 154 | | |
136 | 155 | | |
137 | | - | |
138 | 156 | | |
139 | 157 | | |
140 | 158 | | |
| |||
0 commit comments