Replies: 2 comments
-
|
!vouch @mq1n |
Beta Was this translation helpful? Give feedback.
-
|
Hello @can1357 It was fair concern, I dug into it and it's half right, so I have addressed the half that's real. What it doesn't break: the live session's cache. The session keeps running on the session model; compaction is a side request, and the post-compaction cold start on the next turn happens regardless of which model wrote the summary (so the transcript got rewritten, that's inherent to compaction, not this patch). The local summary path is also unaffected: it sends a serialized conversation under its own system prompt, so it never shared a prefix with the session cache to begin with. What it did break: the compaction request's own cache hit, which is basically the entire cost of a compaction. OpenAI remote compaction (V1 /responses/compact and V2 streaming) replays the session's exact native history; the prefix the provider already has cached, so on the session model most of that input bills at cacheRead ($0.175/M) instead of list ($1.75/M). Prefix caches don't cross models, and spark's list price is identical to codex's, so routing the replay to spark turned a warm ~10x-discounted read into a full-price cold prefill for API-key users. That's a real regression and just figured defaulting into it was wrong. Fixed now as the inferred spark default now only fires when it can't cost anyone money:
Explicit compactionModel / modelRoles.compaction targets bypass the gate so if someone deliberately routes compaction to spark on a metered key, that's their trade-off to make. Added tests for the metered-blocked, subscription-allowed, remote-disabled, zero-cost, explicit-override, and mid-turn cases, and documented the cache economics in docs/compaction.md. Reopened #4689 to replace #4519 (couldn't reopen it after the bot close); same branch, now with the prompt-cache gate from my comment above included and tested. Thanks for the vouch! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I opened a PR for configurable compaction model routing:
#4519
Short version: this lets LLM summary compaction use a dedicated model role (
modelRoles.compaction) while keeping the existing per-modelcompactionModeloverride as the highest-precedence choice.It also adds
compaction.preferFastModelenabled by default, so OpenAI Codex sessions can try a provider-local fast model likegpt-5.3-codex-sparkfor compact summaries, then fall back to the maincompact/active model when the prepared input is too large or needs unsupported modalities.
Happy to adjust the scope or naming if maintainers prefer a different shape. Thanks!
Beta Was this translation helpful? Give feedback.
All reactions