feat(agent): opt-in warm-poke cache-warming driver (#10)#147
Merged
Conversation
Periodically keep active threads' LLM prompt cache hot so the next real message lands on a discounted cache-read instead of a full-price cache-write. A "warm poke" is a tool-less, near-zero-output agent turn enqueued on an active thread. selectWarmPokeTargets gates aggressively on economics: only threads whose cache is currently WARM and about to lapse before the next driver tick, anchored on real activity (last user message, or last run for noHistory cron threads), capped at max_pokes_per_active_period since that anchor. Cold caches are never re-warmed (a write with no guarantee a real message follows), and boundless threads with no live client session are skipped. The driver is opt-in via cache_warming.json (disabled by default) with cadence_ms (< 1h cache TTL), active_window_ms, and max_pokes_per_active_period. It runs from the web host that owns dispatch and enqueues a `cache_warm_poke` notification per target. A poke wakeup is detected in the dispatch runFn (every claimed entry is a cache_warm_poke notification) and runs locally with cacheWarmOnly: tools are disabled (getMergedTools returns undefined) and max_tokens is clamped to WARM_POKE_MAX_OUTPUT_TOKENS on both the local and remote-dispatch inference paths. Pokes never delegate — a delegated poke would rebuild its config on the remote host and run with full tools. The marker prefix on the notification content lets the selector count prior pokes via a content LIKE without a dedicated column. Pokes anchor on real activity only, so a poke turn does not reset the active-period anchor; it does record a turn (cache-read) that extends the TTL and resets the just-in-time clock, so the thread won't be re-poked until it nears expiry again.
…onfig into model_backends.json Two flaws Kara flagged on #147 share one root cause: cadence_ms is a redundant knob. It restated information already carried by each backend's cache_ttl, and as a single global value it was wrong the moment two backends had different TTLs (e.g. a 5m backend always poked cold and paid a full cache-write with no read benefit). - Remove cadence_ms. selectWarmPokeTargets now takes a resolveTtlMs(threadId) callback (server wires it via resolveThreadModel + modelRouter.getCacheTtl) and a fixed scan interval. The just-in-time window is ttl - scanInterval, derived per-thread, so one driver serves 5m and 1h backends correctly. Poke spacing self-regulates: each poke writes a turn that resets the clock. - Null TTL (backend doesn't cache) now skips the thread — nothing to warm. - Fold the surviving config into an optional cache_warming block in model_backends.json, co-located with the per-backend cache_ttl + cache pricing the economics depend on, instead of a standalone cache_warming.json. Trim to the two genuinely-tunable fields: enabled and max_pokes_per_active_period. active_window and scan interval are constants.
The warm-poke per-active-period cap is the load-bearing economic control of the cache-warming driver, and its break-even point varies dramatically by provider: it scales with the cache-write/read price ratio, so a 5m backend with cheap reads tolerates many more pokes per caught arrival than a 1h backend with expensive writes. A single global cap could only ever be right for one backend in a mixed cluster. Move max_pokes_per_active_period from the top-level cache_warming block onto modelBackendSchema, sibling to cache_ttl (whose economics it shares). cache_warming keeps only `enabled` -- the driver master switch. The cap rides the same router seam as cache_ttl (invariant #17): toRouterConfig -> BackendConfig.cacheWarmMaxPokes -> a new ModelRouter.getCacheWarmMaxPokes accessor. It stops at the router -- the cap is never read at inference time, so it does NOT ride ModelResolution / agent-loop / relay-processor. Collapse the selector's two per-thread inputs (resolveTtlMs + a constant cap) into one resolvePokePolicy(threadId) -> {ttlMs, maxPokes} | null, so TTL and cap derive from a single model resolution per thread per scan. Absent field -> DEFAULT_WARM_POKE_MAX_PER_PERIOD (3); 0 -> never warm threads on that backend (clean per-backend opt-out even while the driver is globally enabled), enforced naturally by the existing pokeCount >= maxPokes gate. Also fix two stale README rows: the cache_warming description (cap is now per-backend) and a bogus cache_warming.json config-file row that never existed (the block lives in model_backends.json) and referenced the already-removed cadence_ms knob.
Collapse the cache-warming config so the entire `cache_warming` block
lives on each backend, rather than a global `enabled` switch at the
model_backends top level plus a per-backend `max_pokes_per_active_period`.
The split was an altitude mismatch: once the poke-count cap had to be
per-backend (its break-even varies dramatically by provider's
cache-write/read price ratio), the global `enabled` was the only thing
left up top, deciding *whether* to warm at a different level than the
parameter deciding whether warming is *economical*. Both signals depend
on the same per-backend `cache_ttl` + cache pricing, so they belong
together.
- shared: `cacheWarmingConfigSchema` now carries both `enabled` (default
false) and `max_pokes_per_active_period` (default 3, 0 = never warm);
moved above `modelBackendSchema` and attached as the optional
per-backend `cache_warming`. Removed the top-level `cache_warming` and
the standalone per-backend `max_pokes_per_active_period`.
- llm: `BackendConfig.cacheWarmMaxPokes` -> `cacheWarming?: {enabled,
maxPokes}`; `getCacheWarmMaxPokes` -> `getCacheWarmConfig`.
- cli: `toRouterConfig` threads the nested object; the warm-poke driver
starts iff any backend opts in (`backends.some(b =>
b.cache_warming?.enabled)`) and `resolvePokePolicy` gates each thread
on its own backend's `cache_warming.enabled` before TTL/cap.
- README config row + doc comments updated.
The driver, selector, and `resolvePokePolicy` return shape are
unchanged; this is purely where the opt-in decision and its cap live.
karashiiro
requested changes
Jun 1, 2026
…harden payload parse Responses to PR #147 review comments: - Split `AgentLoopConfig.cacheWarmOnly` into orthogonal `noTools` + `maxOutputTokens`. The single use-case flag conflated tool suppression with the output-token clamp; the two knobs are now independent and the warm-poke caller (server.ts) sets both. `WARM_POKE_MAX_OUTPUT_TOKENS` moves from agent-loop to the caller. - Replace the raw `JSON.parse(...) as { type? }` cast + bare `catch` in the warm-poke detection with `isWarmPokeNotificationPayload`, a Zod `safeParse`-backed predicate in cache-warm-poke.ts. A non-JSON or non-matching payload is simply "not a warm poke" → normal full-tools wakeup; documented as intentional, not a swallowed error. Unit-tested. - Rename config key `cache_warming.max_pokes_per_active_period` → `max_pokes` (the `_per_active_period` was redundant with the doc). - Trim the over-detailed `model_backends.json` exposition in the README config table back to sibling terseness. Lint + typecheck clean; agent + cli suites green (2343 pass).
This was referenced Jun 1, 2026
karashiiro
approved these changes
Jun 1, 2026
karashiiro
pushed a commit
that referenced
this pull request
Jun 4, 2026
The README config-file table is the wrong altitude for per-field detail (#147 had to trim cache_warming back to sibling terseness). Add a real home: docs/config.md documents every field of all ten config files plus persona.md, grounded field-for-field in the Zod schemas in config-schemas.ts. README keeps its one-line-per-file table and points to the new reference. Also documents network.json, which the README table omitted entirely. Closes #150.
karashiiro
pushed a commit
that referenced
this pull request
Jun 16, 2026
AgentLoopConfig.platform was documented as a 'platform identifier when the loop runs in a platform context (e.g. "discord")' - the pre-MCP framing flagged in the PR #147 review. The field is really the originating surface tag (same value space as threads.interface), used for intake-affinity routing (Invariant #15) and cache-TTL/interface derivation; the loop itself is connector-agnostic now that connectors are in-process MCP servers whose tools arrive through the normal tool list. Rewrote the doc comment to say that. relay-processor finalizeProcess only writes the relay response - platform delivery moved to MCP tool calls (discord_send_message) inside the loop, so dropped the stale 'and platform delivery' from its onComplete comment. Closes #151
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.