feat(scheduler): add cache_ttl "never" sentinel for always-warm lanes - #245
feat(scheduler): add cache_ttl "never" sentinel for always-warm lanes#245iceteaSA wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 20 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
Note on the red The v0.33.0 schema added those four leaves without updating the dashboard's |
Lanes kept warm by external keepwarm mechanisms (prewarm proxies, dedicated cache-keep tools) re-warm the provider prompt cache out-of-band, so MC's idle>TTL heuristic false-positives: both TTL consumers (scheduler idle-execute and the ttl_idle m[0] fold) initiate a rebuild believed free that is actually a full paid cache-write (measured 450-560K tokens on large sessions). cache_ttl: "never" (string or per-model value, case-insensitive) disables both consumers: parseCacheTtl returns Infinity, so elapsed>ttl / elapsed>=ttl never fire. Rust scheduler mirrors the sentinel with u64::MAX (predicates already use saturating_sub). Status surfaces render "never expires (always-warm lane)" instead of a bogus countdown: /ctx-status, the TUI sidebar (JSON-safe cacheNeverExpires flag on StatusDetail), and Pi's status dialog (which previously parsed the TTL with a private fallback parser). hardCacheExpired extracted to a pure computeHardCacheExpired helper with direct coverage of the never-chain. Tradeoff documented in CONFIGURATION.md: on a genuinely cold start the free-fold window is not detected on such lanes; mutations then apply at the execute threshold.
…TL diagnostics - rpc-handlers: the cacheNeverExpires branch assigned Infinity to cacheRemainingMs; JSON.stringify converts Infinity to null over RPC, violating the numeric StatusDetail contract. Use 0 and let the cacheNeverExpires flag carry the semantics (the TUI keys on it first). - computeHardCacheExpired: the extraction dropped the invalid-cache-ttl-fallback pass outcome and session log on parse failure. Add an onInvalid callback; the transform call site restores the exact pre-extraction record + log. - test: seed last_response_time in the never-TTL status test — the guarded branch only runs when lastResponseTime > 0, so the assertion was vacuous without it (verified red: Infinity revert now fails it).
620c138 to
222c184
Compare
Brings the cache_ttl "never" sentinel (PR cortexkit#245) into the live union branch so the always-warm lanes can drop the 999h workaround. Resolutions: - rpc-handlers.ts: both imports (getSkillMemoryStats + parseCacheTtl). - rpc-handlers.test.ts / execute-status.test.ts: the PR branch was cut before external-memory made buildStatusDetail and executeStatus async, so its two new tests called them synchronously and asserted against a Promise. Added the awaits; the external-memory describe block also lost its closing braces to the merge and was restored.
Problem
cache_ttlassumes idle > TTL means the provider evicted the prompt cache, so the next prefix rebuild is free. Two consumers act on that assumption:scheduler.ts), andmustMaterializefires thettl_idleHARD fold (viahardCacheExpiredintransform.ts).On lanes kept warm by an external keepwarm mechanism (prewarm proxies, dedicated cache-keep tools that re-warm the provider cache out-of-band), the assumption is wrong: the cache never goes cold, MC's
lastResponseTimegoes stale anyway, and both consumers false-positive. MC then initiates a rebuild it believes is free that is actually a full paid cache-write — measured 450-560K tokens per fold on large sessions. There was no way to express "this lane never goes cold": the config requires a duration.Fix
cache_ttl: "never"(case-insensitive, works as the string form or any per-model value):parseCacheTtl("never")returnsInfinity; both consumers go inert through the existing comparisons (elapsed > Infinity/elapsed >= Infinityare never true) — no new branches in the hot path.u64::MAX(its predicates already usesaturating_sub, so no overflow path)./ctx-statusshowsnever expires (always-warm lane); the TUI sidebar gets a JSON-safecacheNeverExpiresflag onStatusDetail(Infinity does not survive JSON-RPC); Pi's status dialog now uses the sharedparseCacheTtl(it previously used a private fallback parser that would have shown a 5m countdown).hardCacheExpiredis extracted into a purecomputeHardCacheExpiredhelper so the"never" -> Infinity -> falsechain has direct unit coverage instead of only flag-consumption coverage.Docs:
CONFIGURATION.mddescribes the sentinel, what it disables, and the tradeoff — after a genuinely cold start (e.g. the keepwarm process died), the free-fold window is not detected on such lanes; mutations then apply at the execute threshold.Verification
tsc --noEmitclean on bothcomputeHardCacheExpiredtestscheduler.rs's test module (note: I could not runcargo testlocally — the crates workspace has path deps outside this repo; the change is a 3-line early return mirroring the TS logic)build-schema/build-config-docsdrift tests green);check:tui-compiledgreenNeed help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Adds
cache_ttl: "never"to disable the idle‑TTL heuristic on always‑warm lanes, preventing false executes and expensive cache rebuilds. Also fixes status serialization and restores invalid‑TTL diagnostics.New Features
parseCacheTtl("never")returns Infinity (TS) /u64::MAX(Rust) so idle‑execute andttl_idlenever fire;computeHardCacheExpiredexported for consistent checks withonInvalid.cacheNeverExpiresflag; Pi now uses the shared parser; docs and schema updated.Bug Fixes
cacheRemainingMsnumeric (0) when TTL is “never” and keys oncacheNeverExpires.onInvalidcallback.Written for commit 222c184. Summary will update on new commits.
Greptile Summary
Adds and consistently surfaces the always-warm
cache_ttl: "never"configuration.Confidence Score: 5/5
The PR appears safe to merge.
No blocking failures remain.
Important Files Changed
u64::MAXand adds predicate and scheduler coverage.Reviews (3): Last reviewed commit: "fix(scheduler): keep cacheRemainingMs JS..." | Re-trigger Greptile