fix(adapter-utils): default k8s adapters to session rotation — stop context_overflow + OOM (BLO-8827) - #290
Merged
Conversation
…context_overflow + OOM (BLO-8827)
opencode_k8s / claude_k8s run a persisted session across timer wakes. Each wake
re-ingests a large working set (repo files, MCP/gbrain, tool outputs), so the
session's raw input re-inflates to 220-290k tokens (cached reads to 6.5M) faster
than the adapter's lossy `/compact` gate can shrink it — BLO-5679 "compaction
not holding". With no rotation the session climbs until it overflows the model
window (context_overflow), and that giant in-heap payload also drove the 8Gi
OOMs on MulticastEngineer.
Root cause: ADAPTER_SESSION_MANAGEMENT had no entry for claude_k8s/opencode_k8s,
and they aren't in LEGACY_SESSIONED_ADAPTER_TYPES, so resolveSessionCompactionPolicy
fell back to {...DEFAULT, enabled:false} — session rotation was DISABLED by
default for the entire k8s agent fleet (sessionRotated=false on every run,
confirmed in prod). Only the lossy /compact ran, and it can't hold.
Fix: register both k8s adapters with a default policy that enables rotation with
a 150k raw-input ceiling (under the smallest mainstream window we run — claude
200k; gpt-5.5 is larger). evaluateSessionCompaction then rotates to a fresh
session + handoff summary before the window overflows, bounding context (and
memory). Still per-agent tunable via runtimeConfig.heartbeat.sessionCompaction.
The 90k /compact gate was a red herring (it fires on rawInputTokens fine);
lowering it would have over-compacted the fleet without fixing "not holding".
Test (server vitest): opencode_k8s/claude_k8s now resolve to enabled rotation
with a 150k ceiling by default; per-agent runtimeConfig override still wins.
tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
|
@ally review |
…efault - Tighten k8s default-policy tests to full toEqual (all 4 fields) so a fat-fingered maxSessionRuns/maxSessionAgeHours can't silently disable a rotation trigger without failing a test. - Add an enabled:false per-agent override test (verifies the advertised disable escape hatch + partial-merge semantics). - Comment: clarify the 150k ceiling gates NON-cached raw input (rawInputTokens, excludes cached reads), checked per completed wake — so it bounds growth across wakes but doesn't hard-cap a single wake past the window. Drop the hard-coded 8Gi figure (deployment detail that can go stale). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + gstack/codex.
Looks good. Correct, tightly-scoped fix with matching tests. I verified each material claim in the PR description against the source at the head SHA (1d6e7cc).
Verification (what I checked)
- Adapter-type strings are the real values, not a typo. The fix keys
ADAPTER_SESSION_MANAGEMENTon"claude_k8s"/"opencode_k8s". These aren't inAGENT_ADAPTER_TYPES(packages/shared/src/constants.ts) andk8sis an environment driver, so the worry was that realagent.adapterTypeis*_local+ a k8s env (which would make the new keys a silent no-op). Confirmed they are the literaladapterTypevalues: existingserver/src/__tests__/heartbeat-opencode-k8s-timer-no-work.test.ts:74builds an agent withadapterType: "opencode_k8s", andserver/src/services/k8s-job-liveness.ts:5documents "the claude_k8s / opencode_k8s adapters." Fix targets the correct strings. - Root cause holds. Pre-PR, neither type had an
ADAPTER_SESSION_MANAGEMENTentry and neither is inLEGACY_SESSIONED_ADAPTER_TYPES, soresolveSessionCompactionPolicy→getAdapterSessionManagement(...) === null→basePolicy = {...DEFAULT, enabled: fallbackEnabled=false}→ rotation disabled (legacy_fallback). Post-PR it resolves toadapter_defaultwithK8S_AGENT_SESSION_POLICY(enabled:true,maxRawInputTokens:150_000). - End-to-end wiring is live.
evaluateSessionCompaction(heartbeat.ts:4365) callsparseSessionCompactionPolicy(:1953→resolveSessionCompactionPolicy(agent.adapterType,…).policy) and gates at:4382onpolicy.enabled && hasSessionCompactionThresholds(policy). WithmaxRawInputTokens:150_000>0the gate now opens for k8s agents;sessionRotatedis recorded at:10299. - Per-agent override still wins.
policy.maxRawInputTokens = explicitOverride.maxRawInputTokens ?? basePolicy.maxRawInputTokens, so MulticastEngineer's livemaxRawInputTokens:180000is preserved — no regression for the already-mitigated agent. Covered by the third test. - Types valid. All four
SessionCompactionPolicyfields present;nativeContextManagement: "unknown"is a validNativeContextManagementmember and is not consumed by the policy logic (informational only — no conflict with rotation).
Suggestions (1)
- [code]
packages/adapter-utils/src/session-compaction.ts:51— The 150k ceiling gives ~50k headroom under claude's 200k window and bounds cross-wake accumulation. It implicitly assumes a freshly-rotated session's per-wake working set sits comfortably below 150k; if a single post-rotation wake re-ingests near/over that, rotation could fire nearly every wake (thrash) while a single wake could still approach the window. Your deploy-verification step (watchsessionRotated=true+context_overflowstops) should cover this — worth also eyeballing post-rotation peakrawInputTokensto confirm the ceiling leaves enough headroom. Low-confidence; the fleet default is the right call regardless.
Strengths
- Excellent diagnosis-to-fix traceability: the code comment and test docstring both encode the root cause (BLO-8827 / BLO-5679) so future readers understand why k8s adapters rotate.
- Tests assert the three behaviors that matter —
opencode_k8sdefault,claude_k8sdefault, and the per-agent override winning — exercising the real resolver viaparseSessionCompactionPolicy. - Minimal blast radius: only the two k8s adapters change; all other adapters' policies are untouched, and the change is per-agent tunable.
Recommended Action
No blockers. Merge when ready; confirm sessionRotated=true appears and context_overflow stops fleet-wide on deploy (as the PR notes).
kkroo
added a commit
that referenced
this pull request
Jun 4, 2026
PR #290 gave opencode_k8s/claude_k8s a 150k raw-input rotation ceiling but its review flagged a gap: parseSessionCompactionPolicy is well-tested (the default IS 150k), yet nothing tests that the *consumer* actually rotates at that ceiling. evaluateSessionCompaction is a non-exported closure over `db` in the heartbeat factory, so the decision wiring (the >= comparator, the non-cached rawInputTokens field, trigger precedence) had no unit coverage — a regression there would pass every policy-shape test and silently stop rotating, the exact BLO-8827 failure mode. Extract the pure rotation-trigger decision into an exported module-scope computeSessionCompactionReason({policy, runsCount, latestRawInputTokens, sessionAgeHours}) and have evaluateSessionCompaction call it. Behavior- preserving: identical reason strings, identical runs→raw→age precedence, identical inclusive >= comparison. Add 9 tests driven by the REAL resolved policy (parseSessionCompactionPolicy, not a hand-built literal, so default and decision can't drift): rotates at exactly 150k (inclusive), not at 149,999; both k8s adapters; null/cache-heavy low raw input does not rotate; the per-agent 180k override shifts the boundary end-to-end; runs/age precedence; and adapter-managed zero-threshold agents never rotate. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (fleet-wide, evidenced in prod)
opencode_k8s/claude_k8srun a persisted session across timer wakes. Each wake re-ingests a large working set (repo files, MCP/gbrain, tool outputs), so the session'srawInputTokensre-inflates to 220–290k (cached reads up to 6.5M, observed on MulticastEngineer) faster than the adapter's lossy/compactcan shrink it — the historical BLO-5679 "session compaction not holding." With no rotation the session climbs until it overflows the model window (context_overflow), and that giant in-heap payload also drove the 8Gi OOMs.Why rotation never fired:
ADAPTER_SESSION_MANAGEMENThad no entry forclaude_k8s/opencode_k8s, and they aren't inLEGACY_SESSIONED_ADAPTER_TYPES, soresolveSessionCompactionPolicyfell back to{...DEFAULT, enabled:false}→ session rotation was disabled by default for the entire k8s agent fleet (sessionRotated=falseon every run, confirmed in prod data).The
90k/compactgate is a red herring — it fires onrawInputTokenscorrectly; lowering it would have over-compacted the whole fleet (lossy: a prior compaction went 170k→3k) without fixing "not holding."Fix
Register both k8s adapters in
ADAPTER_SESSION_MANAGEMENTwith a default policy that enables rotation with a 150k raw-input ceiling — under the smallest mainstream window we run (claude 200k; gpt-5.5 is larger).evaluateSessionCompactionthen rotates to a fresh session + handoff summary before the window overflows, bounding context (and memory). Still per-agent tunable viaruntimeConfig.heartbeat.sessionCompaction.Test (server vitest)
opencode_k8s/claude_k8snow resolve to rotation-enabled with a 150k ceiling by default; a per-agentruntimeConfigoverride still wins.tsc --noEmitclean.Context
Sibling to #283 (image-bump deadlock + Job-identity) and #287 (updatedAt-churn reaper). MulticastEngineer already has the equivalent per-agent config (16Gi memory +
maxRawInputTokens:180000) applied live as the immediate mitigation; this makes rotation the fleet default so every k8s agent is protected. On deploy, verifysessionRotated=truebegins appearing andcontext_overflowstops fleet-wide.🤖 Generated with Claude Code