Skip to content

fix(adapter-utils): default k8s adapters to session rotation — stop context_overflow + OOM (BLO-8827) - #290

Merged
kkroo merged 2 commits into
masterfrom
omar/k8s-adapters-session-rotation-default
Jun 4, 2026
Merged

fix(adapter-utils): default k8s adapters to session rotation — stop context_overflow + OOM (BLO-8827)#290
kkroo merged 2 commits into
masterfrom
omar/k8s-adapters-session-rotation-default

Conversation

@kkroo

@kkroo kkroo commented Jun 4, 2026

Copy link
Copy Markdown

Root cause (fleet-wide, evidenced in prod)

opencode_k8s / claude_k8s run a persisted session across timer wakes. Each wake re-ingests a large working set (repo files, MCP/gbrain, tool outputs), so the session's rawInputTokens re-inflates to 220–290k (cached reads up to 6.5M, observed on MulticastEngineer) faster than the adapter's lossy /compact can shrink it — the historical BLO-5679 "session compaction not holding." With no rotation the session climbs until it overflows the model window (context_overflow), and that giant in-heap payload also drove the 8Gi OOMs.

Why rotation never fired: ADAPTER_SESSION_MANAGEMENT had no entry for claude_k8s/opencode_k8s, and they aren't in LEGACY_SESSIONED_ADAPTER_TYPES, so resolveSessionCompactionPolicy fell back to {...DEFAULT, enabled:false}session rotation was disabled by default for the entire k8s agent fleet (sessionRotated=false on every run, confirmed in prod data).

The 90k /compact gate is a red herring — it fires on rawInputTokens correctly; lowering it would have over-compacted the whole fleet (lossy: a prior compaction went 170k→3k) without fixing "not holding."

Fix

Register both k8s adapters in ADAPTER_SESSION_MANAGEMENT with a default policy that enables rotation with a 150k raw-input ceiling — under the smallest mainstream window we run (claude 200k; gpt-5.5 is larger). evaluateSessionCompaction then rotates to a fresh session + handoff summary before the window overflows, bounding context (and memory). Still per-agent tunable via runtimeConfig.heartbeat.sessionCompaction.

Test (server vitest)

opencode_k8s/claude_k8s now resolve to rotation-enabled with a 150k ceiling by default; a per-agent runtimeConfig override still wins. tsc --noEmit clean.

Context

Sibling to #283 (image-bump deadlock + Job-identity) and #287 (updatedAt-churn reaper). MulticastEngineer already has the equivalent per-agent config (16Gi memory + maxRawInputTokens:180000) applied live as the immediate mitigation; this makes rotation the fleet default so every k8s agent is protected. On deploy, verify sessionRotated=true begins appearing and context_overflow stops fleet-wide.

🤖 Generated with Claude Code

…context_overflow + OOM (BLO-8827)

opencode_k8s / claude_k8s run a persisted session across timer wakes. Each wake
re-ingests a large working set (repo files, MCP/gbrain, tool outputs), so the
session's raw input re-inflates to 220-290k tokens (cached reads to 6.5M) faster
than the adapter's lossy `/compact` gate can shrink it — BLO-5679 "compaction
not holding". With no rotation the session climbs until it overflows the model
window (context_overflow), and that giant in-heap payload also drove the 8Gi
OOMs on MulticastEngineer.

Root cause: ADAPTER_SESSION_MANAGEMENT had no entry for claude_k8s/opencode_k8s,
and they aren't in LEGACY_SESSIONED_ADAPTER_TYPES, so resolveSessionCompactionPolicy
fell back to {...DEFAULT, enabled:false} — session rotation was DISABLED by
default for the entire k8s agent fleet (sessionRotated=false on every run,
confirmed in prod). Only the lossy /compact ran, and it can't hold.

Fix: register both k8s adapters with a default policy that enables rotation with
a 150k raw-input ceiling (under the smallest mainstream window we run — claude
200k; gpt-5.5 is larger). evaluateSessionCompaction then rotates to a fresh
session + handoff summary before the window overflows, bounding context (and
memory). Still per-agent tunable via runtimeConfig.heartbeat.sessionCompaction.

The 90k /compact gate was a red herring (it fires on rawInputTokens fine);
lowering it would have over-compacted the fleet without fixing "not holding".

Test (server vitest): opencode_k8s/claude_k8s now resolve to enabled rotation
with a 150k ceiling by default; per-agent runtimeConfig override still wins.
tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kkroo

kkroo commented Jun 4, 2026

Copy link
Copy Markdown
Author

@ally review

…efault

- Tighten k8s default-policy tests to full toEqual (all 4 fields) so a
  fat-fingered maxSessionRuns/maxSessionAgeHours can't silently disable a
  rotation trigger without failing a test.
- Add an enabled:false per-agent override test (verifies the advertised
  disable escape hatch + partial-merge semantics).
- Comment: clarify the 150k ceiling gates NON-cached raw input (rawInputTokens,
  excludes cached reads), checked per completed wake — so it bounds growth
  across wakes but doesn't hard-cap a single wake past the window. Drop the
  hard-coded 8Gi figure (deployment detail that can go stale).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@blockcast-ci-packages blockcast-ci-packages Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + gstack/codex.

Looks good. Correct, tightly-scoped fix with matching tests. I verified each material claim in the PR description against the source at the head SHA (1d6e7cc).

Verification (what I checked)

  • Adapter-type strings are the real values, not a typo. The fix keys ADAPTER_SESSION_MANAGEMENT on "claude_k8s" / "opencode_k8s". These aren't in AGENT_ADAPTER_TYPES (packages/shared/src/constants.ts) and k8s is an environment driver, so the worry was that real agent.adapterType is *_local + a k8s env (which would make the new keys a silent no-op). Confirmed they are the literal adapterType values: existing server/src/__tests__/heartbeat-opencode-k8s-timer-no-work.test.ts:74 builds an agent with adapterType: "opencode_k8s", and server/src/services/k8s-job-liveness.ts:5 documents "the claude_k8s / opencode_k8s adapters." Fix targets the correct strings.
  • Root cause holds. Pre-PR, neither type had an ADAPTER_SESSION_MANAGEMENT entry and neither is in LEGACY_SESSIONED_ADAPTER_TYPES, so resolveSessionCompactionPolicygetAdapterSessionManagement(...) === nullbasePolicy = {...DEFAULT, enabled: fallbackEnabled=false} → rotation disabled (legacy_fallback). Post-PR it resolves to adapter_default with K8S_AGENT_SESSION_POLICY (enabled:true, maxRawInputTokens:150_000).
  • End-to-end wiring is live. evaluateSessionCompaction (heartbeat.ts:4365) calls parseSessionCompactionPolicy (:1953resolveSessionCompactionPolicy(agent.adapterType,…).policy) and gates at :4382 on policy.enabled && hasSessionCompactionThresholds(policy). With maxRawInputTokens:150_000>0 the gate now opens for k8s agents; sessionRotated is recorded at :10299.
  • Per-agent override still wins. policy.maxRawInputTokens = explicitOverride.maxRawInputTokens ?? basePolicy.maxRawInputTokens, so MulticastEngineer's live maxRawInputTokens:180000 is preserved — no regression for the already-mitigated agent. Covered by the third test.
  • Types valid. All four SessionCompactionPolicy fields present; nativeContextManagement: "unknown" is a valid NativeContextManagement member and is not consumed by the policy logic (informational only — no conflict with rotation).

Suggestions (1)

  • [code] packages/adapter-utils/src/session-compaction.ts:51 — The 150k ceiling gives ~50k headroom under claude's 200k window and bounds cross-wake accumulation. It implicitly assumes a freshly-rotated session's per-wake working set sits comfortably below 150k; if a single post-rotation wake re-ingests near/over that, rotation could fire nearly every wake (thrash) while a single wake could still approach the window. Your deploy-verification step (watch sessionRotated=true + context_overflow stops) should cover this — worth also eyeballing post-rotation peak rawInputTokens to confirm the ceiling leaves enough headroom. Low-confidence; the fleet default is the right call regardless.

Strengths

  • Excellent diagnosis-to-fix traceability: the code comment and test docstring both encode the root cause (BLO-8827 / BLO-5679) so future readers understand why k8s adapters rotate.
  • Tests assert the three behaviors that matter — opencode_k8s default, claude_k8s default, and the per-agent override winning — exercising the real resolver via parseSessionCompactionPolicy.
  • Minimal blast radius: only the two k8s adapters change; all other adapters' policies are untouched, and the change is per-agent tunable.

Recommended Action

No blockers. Merge when ready; confirm sessionRotated=true appears and context_overflow stops fleet-wide on deploy (as the PR notes).

@kkroo
kkroo merged commit f10cf03 into master Jun 4, 2026
13 checks passed
kkroo added a commit that referenced this pull request Jun 4, 2026
PR #290 gave opencode_k8s/claude_k8s a 150k raw-input rotation ceiling but
its review flagged a gap: parseSessionCompactionPolicy is well-tested (the
default IS 150k), yet nothing tests that the *consumer* actually rotates at
that ceiling. evaluateSessionCompaction is a non-exported closure over `db`
in the heartbeat factory, so the decision wiring (the >= comparator, the
non-cached rawInputTokens field, trigger precedence) had no unit coverage —
a regression there would pass every policy-shape test and silently stop
rotating, the exact BLO-8827 failure mode.

Extract the pure rotation-trigger decision into an exported module-scope
computeSessionCompactionReason({policy, runsCount, latestRawInputTokens,
sessionAgeHours}) and have evaluateSessionCompaction call it. Behavior-
preserving: identical reason strings, identical runs→raw→age precedence,
identical inclusive >= comparison.

Add 9 tests driven by the REAL resolved policy (parseSessionCompactionPolicy,
not a hand-built literal, so default and decision can't drift): rotates at
exactly 150k (inclusive), not at 149,999; both k8s adapters; null/cache-heavy
low raw input does not rotate; the per-agent 180k override shifts the boundary
end-to-end; runs/age precedence; and adapter-managed zero-threshold agents
never rotate.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant