Skip to content

fix(proxy): suppress Claude subscription on ctx in baseline failover exhaustion check#533

Open
rohith500 wants to merge 2 commits into
workweave:mainfrom
rohith500:fix/baseline-failover-subscription-billing-race
Open

fix(proxy): suppress Claude subscription on ctx in baseline failover exhaustion check#533
rohith500 wants to merge 2 commits into
workweave:mainfrom
rohith500:fix/baseline-failover-subscription-billing-race

Conversation

@rohith500

Copy link
Copy Markdown
Contributor

Problem

When a Claude subscription is healthy at the pre-dispatch exhaustion check (service.go:1794) but becomes exhausted during the primary OSS model dispatch (concurrent request burns the plan window), the baseline failover path has a race that causes the deployment key cost to be absorbed silently with no billing record.

Execution trace of the race:

Line Action ctx suppressed?
1794 claudeSubscriptionExhausted = false → suppression NOT applied to ctx No
1797 resolveAndInjectCredentials(ctx, ProviderOSS) — OSS credentials set No
2051 Primary OSS dispatch fails; concurrent request flips snapshot to exhausted No
2097 claudeSubscriptionExhausted = true (snapshot now exhausted) No
2099 baselineCtx = withSuppressedClaudeSubscription(baselineCtx) baselineCtx: Yes, ctx: No
2101 resolveAndInjectCredentials(baselineCtx, ProviderAnthropic) → deployment key ✓
2109 Baseline dispatch succeeds on Weave's deployment key
2147 ctx = resolveAndInjectCredentials(ctx, ProviderAnthropic)subscription token injected No ← bug
2200 servedOnSubscription(ctx)truedelta = 0deployment key cost unrecorded

The deployment key pays for the turn, but the ledger records a zero-debit row (subscription rate). Three attributes are misattributed:

  • cost.subscription_served OTel attribute (service.go:2200) — reports true instead of false
  • SubscriptionServed billing field (service.go:2931) — causes delta = 0 instead of full cost
  • credentialKeyPrefix/Suffix/credSource telemetry (service.go:2229) — attributes the turn to the subscription token instead of the deployment key

Fix

Apply withSuppressedClaudeSubscription to ctx alongside baselineCtx at the baseline exhaustion check:

// BEFORE:
if s.claudeSubscriptionExhausted(ctx, r.Header) {
        baselineCtx = withSuppressedClaudeSubscription(baselineCtx)
}
 
// AFTER:
if s.claudeSubscriptionExhausted(ctx, r.Header) {
        ctx = withSuppressedClaudeSubscription(ctx)
        baselineCtx = withSuppressedClaudeSubscription(baselineCtx)
}

With the fix, ctx carries the suppression flag through to the re-resolution at line 2147, which falls through to the deployment key. All three misattributed fields are corrected by the same change.

Safety properties verified:

  • Happy path (no exhaustion): the new line sits inside three nested conditions requiring exhaustion + failed primary + baseline eligible — pure no-op on every successful turn
  • Double-suppression: withSuppressedClaudeSubscription applied twice is idempotent — context.WithValue wraps immutably, .Value() returns true either way
  • ProxyOpenAIChatCompletion: no baseline failover path exists there — the fix is complete

Verification

TestResolveAndInjectCredentials_BaselineFailoverRace in subscription_exhaustion_internal_test.go proves both paths through resolveAndInjectCredentials and servedOnSubscription:

  • Buggy path: unsuppressed ctx → subscription token injected → servedOnSubscription = true
  • Fixed path: suppressed ctx → falls through to deployment key → servedOnSubscription = false
    make check — all 30 packages pass.

@greptile-apps

greptile-apps Bot commented Jun 27, 2026

Copy link
Copy Markdown

PR author is not in the allowed authors list.

@steventohme steventohme left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for chasing this down — but I think this one-liner is a no-op as written and doesn't actually fix the billing miscount. The bug is real; this just doesn't move it.

Why it's a no-op

servedOnSubscription(ctx) reads CredentialsFromContext(ctx).OAuth — the credential stamped by resolveAndInjectCredentials, not the suppression marker:

func servedOnSubscription(ctx context.Context) bool {
	creds := CredentialsFromContext(ctx)
	return creds != nil && creds.OAuth
}

withSuppressedClaudeSubscription sets a separate context key (suppressClaudeSubscriptionContextKey), and that marker only takes effect on the next resolveAndInjectCredentials call. In ProxyMessages, ctx's credential is set exactly once — at line 1797. After the new line at 2097 there is no ctx = resolveAndInjectCredentials(ctx, …); only baselineCtx is re-resolved and dispatched. (Grepping the file, the only ctx = resolveAndInjectCredentials calls are 1797 and 3302, the latter in a different handler.)

So marking ctx suppressed at 2097 leaves the OAuth credential injected at 1797 untouched → servedOnSubscription(ctx) at line 2192 (cost.subscription_served) and line 2917 (SubscriptionServed) still returns true, and the deployment-key cost still goes unrecorded.

Why the test passes anyway

TestResolveAndInjectCredentials_BaselineFailoverRace manually calls resolveAndInjectCredentials(ctxFixed, …) after suppression:

ctxFixed := withSuppressedClaudeSubscription(ctx)
outFixed := resolveAndInjectCredentials(ctxFixed, providers.ProviderAnthropic, headers) // production never does this after 2097

That proves the helper's behavior, not the integration. The production path never re-resolves ctx after 2097, so the assertion that's green here doesn't reflect what ProxyMessages actually does.

Note on the line numbers

The issue's trace references ctx = resolveAndInjectCredentials(ctx, ProviderAnthropic) at "line 2147", but current main resolves baselineCtx, not ctx — the baseline path was refactored to a separate variable since the issue was filed, which is what makes this line inert.

Suggested fix

One of:

  1. (preferred) After a successful baseline failover, set ctx = baselineCtx, so every ctx-derived downstream attribute (billing SubscriptionServed, cost.subscription_served, and the credential prefix/suffix/source telemetry) reflects the credential that actually served the turn — not just billing.
  2. Re-resolve right after the suppression line: ctx = resolveAndInjectCredentials(ctx, providers.ProviderAnthropic, r.Header).

And the test should drive ProxyMessages (or at minimum assert on the post-baseline ctx), so it would fail against the current one-line change.

TestResolveAndInjectCredentials_BaselineFailoverRace tested the
resolveAndInjectCredentials helper in isolation by calling it manually
after suppression — a call production never makes at that point in
ProxyMessages. It passed whether or not line 2098 existed in service.go
and did not discriminate the bug from the fix.

Replace with TestBaselineFailoverCtxSuppression which simulates the
actual production chain: context.Background() as the starting ctx (OSS
primary at line 1797 left no Anthropic credential on ctx), suppression
applied or not at line 2098, re-resolution at line 2148 with
ProviderAnthropic.

fails_without_fix: no suppression on ctx before re-resolution ->
resolveAndInjectCredentials injects OAuth -> servedOnSubscription = true
(documents the bug).

passes_with_fix: withSuppressedClaudeSubscription applied before
re-resolution -> both guards in resolveAndInjectCredentials fire (line
2694 block skipped, line 2759 bearer nil'd) -> servedOnSubscription =
false (proves the fix).

Removing withSuppressedClaudeSubscription from passes_with_fix makes it
go red -- the test correctly discriminates the fixed from the broken path.

Signed-off-by: N Rohith Reddy <rohithreddy2202@gmail.com>
…exhaustion check

When claudeSubscriptionExhausted fires at the baseline check (line 2097)
but not at the pre-dispatch check (line 1794), ctx carries an OAuth
credential from OSS resolution at line 1797. The re-resolution at line
2148 (ctx = resolveAndInjectCredentials(ctx, finalProvider, r.Header))
then re-injects the OAuth token -- causing servedOnSubscription(ctx) to
return true even though Weave's deployment key served the turn. Three
misattributions result:
  - cost.subscription_served OTel attribute: true instead of false
  - SubscriptionServed billing field: delta = 0 instead of full cost
  - credentialKeyParts telemetry: subscription token instead of deployment key

Fix: withSuppressedClaudeSubscription(ctx) at line 2098 alongside
baselineCtx. The suppression marker survives untouched to line 2148
(ctx is not reassigned between 2098 and 2148). resolveAndInjectCredentials
skips the OAuth block (line 2694: !suppressClaudeSub guard) and drops
the OAuth bearer on the non-router-keyed path (line 2759 guard). All
three downstream reads of ctx are corrected.

Signed-off-by: N Rohith Reddy <rohithreddy2202@gmail.com>
@rohith500 rohith500 force-pushed the fix/baseline-failover-subscription-billing-race branch from cbe4d87 to a9c6e95 Compare June 29, 2026 17:38
@rohith500

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review — you're right that the test was bad and I've fixed it. But I want to flag what I think is a factual error in the "no-op" diagnosis.

Line 2148 exists

grep -n "ctx = resolveAndInjectCredentials" internal/proxy/service.go
1797:   ctx = resolveAndInjectCredentials(ctx, decision.Provider, r.Header)
2148:   ctx = resolveAndInjectCredentials(ctx, finalProvider, r.Header)
3317:   ctx = resolveAndInjectCredentials(ctx, decision.Provider, r.Header)
3579:   ctx = resolveAndInjectCredentials(ctx, finalProvider, r.Header)

Your review says "the only ctx = resolveAndInjectCredentials calls are 1797 and 3302" — but 2148 is there on current main. Its own comment says:

Update ctx so credentialKeyParts reads the correct key parts for telemetry.

This is the re-resolution the fix depends on.

Why the fix is not a no-op

With line 2098 in place, the chain is:

Line What happens
2096 baselineCtx := ctx
2098 ctx = withSuppressedClaudeSubscription(ctx) — suppression marker set
2099–2123 ctx is not reassigned — only baselineCtx, decision, bindings, baselineAttempted, baselineFailoverUsed change (verified line by line)
2148 ctx = resolveAndInjectCredentials(ctx, finalProvider, r.Header) — re-resolves with suppression present
2694 if provider == ProviderAnthropic && !suppressClaudeSub — false, OAuth block skipped entirely
2759 if suppressClaudeSub && provider == ProviderAnthropic && client != nil && client.OAuth { client = nil } — second guard drops the OAuth bearer on the non-router-keyed path
2200, 2932 servedOnSubscription(ctx)false

Where you're right — the test

The old TestResolveAndInjectCredentials_BaselineFailoverRace tested the helper in isolation. It called resolveAndInjectCredentials(ctxFixed, ...) manually after suppression — something production never does after line 2097. It passed whether or not line 2098 existed in service.go and did not discriminate the bug from the fix.

The new TestBaselineFailoverCtxSuppression starts from context.Background() (correctly simulating the OSS-primary scenario — no Anthropic credential on ctx after line 1797):

  • fails_without_fix: no suppression → resolveAndInjectCredentials injects OAuth → servedOnSubscription = true (documents the bug)
  • passes_with_fix: withSuppressedClaudeSubscription applied → both guards fire → servedOnSubscription = false (proves the fix)

Removing withSuppressedClaudeSubscription from passes_with_fix makes it go red — confirmed by running the test with the call removed. It correctly discriminates the fixed from the broken path.

Force-pushed.

@rohith500 rohith500 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detailed response left as a PR comment above — the short version: line 2148
(ctx = resolveAndInjectCredentials(ctx, finalProvider, r.Header)) exists on
current main and was missed in the grep. The suppression marker from line 2098
flows through to that re-resolution, which is what makes the fix effective.
Test rewritten and force-pushed.

@rohith500 rohith500 requested a review from steventohme July 1, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants