feat(sts): cache AssumeRoleWithWebIdentity responses across isolates

[alukach]

Draft / stacked on fix/sts-request-timeout (#172). The post_form change
builds on that PR's timeout, and the two are complementary (cache removes STS
from the hot path; timeout bounds the rare cold miss). Retarget to main once
#172 merges.

Problem

Private products federate to AWS STS (AssumeRoleWithWebIdentity) on the cold path. multistore's credential cache lives in per-isolate memory (OIDC_PROVIDER is a OnceLock), and Cloudflare spins up many short-lived isolates — so a large fraction of requests re-run the STS exchange on the request hot path. When that exchange stalls, the worker hangs until the edge kills it, surfacing to the app as an unparseable 503. This is the root cause behind the intermittent "product won't load, self-heals on reload" reports.

Approach

Add an L2 cache for the STS response, shared across isolates within a colo via the Cloudflare Cache API — the same pattern already used for Source API responses in source_api/cache.rs. It sits under multistore's in-isolate L1 cache:

• L1 (multistore, per-isolate): caches typed BackendCredentials, single-flights within an isolate.
• L2 (this PR, per-colo): caches the raw STS response body, keyed by RoleArn (L1's own cache key). On a hit, the proxy skips the STS round-trip entirely.

The only seam data.source.coop controls in the mint path is FetchHttpExchange::post_form (the outbound STS call) — get_credentials and the L1 cache live inside multistore. So the L2 cache wraps post_form.

Effect: STS goes from ~once per isolate per credential lifetime → ~once per colo. The slow exchange leaves the user hot path almost entirely.

What's cached (and not)

• Only AssumeRoleWithWebIdentity forms (role_arn_from_form returns None for other actions / Azure-GCP flows → bypass).
• TTL = time to the response's <Expiration> minus a 300s lead (≥ multistore's 60s refresh lead, so an L2 entry always expires before L1 would call the derived credential stale).
• STS error documents are never cached — ttl_secs returns None when there's no parseable <Expiration>.
• On an L2 hit we still mint the (cheap, local) JWT; only the slow STS network call is skipped. Skipping the mint too would need an L1-level hook in multistore (noted below).

Security

The cached values are short-lived, role-scoped temporary credentials, stored under a synthetic non-routable cache key (https://sts-creds.cache.internal/…, never a real edge request URL, so not externally addressable), with TTL ≤ credential lifetime, per-colo. If a deployment needs global reach, encryption-at-rest, or true cross-isolate single-flight (a cold colo can still see a small STS herd), the same cache_key/ttl_secs helpers drop into KV (global, encrypted) or a Durable Object (global, single-flight).

Follow-up (not here)

Cleaner long-term: give multistore's CredentialCache::get_or_fetch an optional runtime L2 hook (the crate doc already anticipates "a runtime can layer an additional cache tier inside the closure"). That caches typed creds at L1 and skips the JWT mint on hits too — but it's a cross-repo API change + release, vs. this which ships from data.source.coop today.

Verification

• cargo test --test sts_cache — 8/8 (role/key/ttl helpers, incl. error-doc and near-expiry → not cached).
• cargo check --target wasm32-unknown-unknown — clean.
• cargo clippy --target wasm32-unknown-unknown -- -D warnings — clean.

🤖 Generated with Claude Code

[claude]

Claude finished @alukach's task in 3m 51s —— View job

---

Review

• Gather context (diff, CLAUDE.md, changed files)
• Review src/sts_cache.rs and src/lib.rs changes
• Review test coverage in tests/sts_cache.rs
• Post final review

✅ No blocking issues — safe to merge.

The implementation is careful and consistent with the existing source_api/cache.rs Cache-API pattern: fail-closed key derivation (cache_inputs_from_form requires both RoleArn and RoleSessionName), a synthetic non-routable cache key, and a TTL helper that only returns Some for a parseable, non-near-expiry <Expiration> — so STS error documents are structurally excluded from the cache. Both the security rationale (keying on session name, not just role) and its known limitation (lossy truncation in sts_session_name()) are already called out in-line with an upgrade path, which is exactly the right way to carry that tradeoff forward.

A couple of non-blocking observations, not required for merge:

• src/sts_cache.rs:20 — REFRESH_LEAD_SECS = 300 is documented as "≥ multistore's own 60s in-isolate refresh lead," but that's a cross-crate invariant this repo can't enforce at compile time. If multistore ever widens its refresh lead past 300s, the safety margin silently erodes. Worth a one-line note-to-self (or a comment in the multistore changelog check) if that dependency is bumped in the future.
• src/sts_cache.rs:47 — cache_inputs_from_form takes the first match for RoleArn/RoleSessionName via .find(...). This is harmless given form is built internally by trusted code with no duplicate keys today, but if that ever changes, the cache key could silently diverge from what's actually sent to STS (which encodes all entries via .form(form)). Not actionable now, just worth keeping in mind if form construction ever becomes more dynamic.
• src/lib.rs:140-143 — sts_cache_get treats any Cache API error as a plain miss (vs. source_api/cache.rs's cached_fetch, which propagates cache-get errors as ProxyError::Internal). That's the right call here specifically because this cache sits in the hot path purely as a latency optimization — degrading to a live STS call on any hiccup is strictly safer than failing the request, and it's called out explicitly in the doc comment.

Test coverage (tests/sts_cache.rs) is solid for the pure helpers: assume-role gating, missing-field fail-closed behavior, key encoding/collision-avoidance, and TTL edge cases (near-expiry, expired, error document, unparseable <Expiration>).

[github-actions]

🚀 Latest commit deployed to https://source-data-proxy-pr-175.source-coop.workers.dev

• Date: 2026-07-14T14:50:41Z
• Commit: b387a10