Route Claude jobs back through ccrotate

[kkroo]

Thinking Path

• Paperclip is the app Blockcast uses to manage AI agents and run their work through Kubernetes-backed adapters.
• The Blockcast deployment seeds provider env vars into paperclip-api and the paperclip StatefulSet, and the Claude K8s adapter inherits those values into agent Job pods.
• Recent BLO/PEN Claude agent runs were failing with empty or malformed HTTP 200 responses on the Anthropic path after the Penstock gateway cutover.
• A live mitigation confirmed that in-cluster ccrotate-serve is serving Claude messages streams successfully while OpenAI/Codex can remain on Penstock.
• This pull request makes that live mitigation durable in Helm and locks it with a render test.
• The benefit is restoring Claude agent liveness without rolling back the OpenAI/Codex Penstock route.

Linked Issues or Issue Description

Refs #502, #451, #449, #450.

What happened

BLO/PEN Claude K8s agent runs repeatedly failed with claude_transient_upstream caused by an empty or malformed HTTP 200 response on the Anthropic gateway path. Some Claude hooks also could not find the shared ccrotate wrapper because /paperclip/bin was not in the inherited PATH.

Expected behavior

Paperclip-spawned Claude jobs should be able to call Anthropic through a healthy in-cluster route, and Claude hooks should be able to resolve the shared ccrotate wrapper.

Steps to reproduce

Run a BLO/PEN claude_k8s heartbeat while the deployment-level Anthropic env points at the Penstock Anthropic gateway, then inspect failed heartbeat runs for malformed HTTP 200 errors.

Paperclip version or commit

Blockcast deployment at 3e52b21ac34d1e82b7e537141ec19aafe981aed3.

Deployment mode

Blockcast Kubernetes Helm deployment.

What Changed

• Routes ANTHROPIC_BASE_URL back to in-cluster http://ccrotate-serve.paperclip.svc:4001 for Blockcast Paperclip pods.
• Points ANTHROPIC_AUTH_TOKEN and ANTHROPIC_API_KEY at paperclip-ccrotate-serve-secrets/serveToken.
• Keeps OpenAI/Codex env vars on the Penstock gateway.
• Adds /paperclip/bin to PATH so Claude hooks can find the shared ccrotate wrapper.
• Updates the Helm render test to assert the split provider route.

Verification

• node deploy/helm/paperclip/tests/runtime-cache.test.mjs
• git diff --check
• Live hotfix applied to deploy/paperclip-api and sts/paperclip; new env confirms Anthropic on ccrotate, OpenAI on Penstock, and /paperclip/bin in PATH.

Risks

• Anthropic traffic is temporarily back on ccrotate, so Penstock Anthropic gateway coverage is reduced until the malformed-200 gateway issue is fixed.
• OpenAI/Codex traffic remains on Penstock, so this is a split route that operators need to keep in mind during the next provider cutover.
• Low Kubernetes rollout risk: this only changes env vars consumed by newly spawned agent jobs.

Model Used

OpenAI GPT-5 Codex coding agent with shell, Kubernetes, GitHub CLI, and repository editing tools.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: code review + test assertions + silent failure analysis.

Critical Issues (1)

• [native-codex] values.blockcast.yaml:254-255 — optional: true removed from paperclip-ccrotate-serve-secrets secret references
  • Impact: Deployment will crash hard (CreateContainerConfigError) if the secret doesn't exist, rather than gracefully degrading as before. The PR description does not confirm this secret is pre-created in the Blockcast cluster.
  • Recommendation: Either restore optional: true (fail-open), document the prerequisite clearly in the comment block (it's missing today), or ensure the secret creation is a blocking prerequisite in the deployment runbook.

Important Issues (1)

• [native-codex] runtime-cache.test.mjs:136-145 + values.blockcast.yaml — Test and values don't verify the secret actually exists
  • Impact: The test assertCcrotateServeSecretEnv checks that Helm renders a reference to paperclip-ccrotate-serve-secrets, but does not verify the secret is present in the cluster or that the key serveToken has a value. A malformed or missing secret will only fail at pod startup, not at Helm render time.
  • Recommendation: Add a comment to the test noting this limitation, or add a prerequisite assertion in the values comment block confirming the secret must exist before deploy (see Critical Issues prerequisite). The PR body mentions "Live hotfix applied" — confirm the secret is now persistent in the deployment.

Suggestions (2)

• [native-codex] values.blockcast.yaml:230 — Removed ROLLBACK comment

  • The old comment explained exactly how to revert if Penstock is restored. The new comment says "temporarily" but doesn't include rollback steps. Consider adding a one-liner rollback guide: # ROLLBACK: restore Penstock Anthropic by reverting ANTHROPIC_BASE_URL, secret name, and key.

• [native-codex] values.blockcast.yaml:251-253 — PREREQUISITES section missing ccrotate secret

  • The PREREQUISITES list items 1 and 2 still reference Penstock setup. Add:

    #   0. Secret `paperclip-ccrotate-serve-secrets` (key `serveToken`) exists in ns paperclip
    #      -- the ccrotate serve auth token.
    

  • This clarifies that the ccrotate route is the blocking dependency (not Penstock).

Strengths

• Clear intent: The test name and comments clearly explain why Anthropic is temporarily back on ccrotate (Penstock malformed-200 issue).
• Split routing preserved: OpenAI/Codex remains on Penstock, reducing the scope of the rollback — good operational hygiene.
• PATH fix included: Adding /paperclip/bin to PATH solves the Claude hooks lookup issue in a single PR.
• Test updates are thorough: All env assertions updated and function renamed to reflect ccrotate routing; no stale assertions left.

Recommended Action

1. Before merge: Address Critical Issue #1 — either restore optional: true OR confirm the secret prerequisite in the comment block AND the deployment runbook. Unclear secret setup is a common deployment-time surprise.
2. This cycle: Add ccrotate secret to PREREQUISITES list (Suggestion #2).
3. Consider: Add rollback instructions comment (Suggestion #1) so operators know exactly how to revert if Penstock is restored.

---

reviewed head: de98f1a