Skip to content

[STG-2749] Surface cache metadata on act/observe/extract results - #2543

Merged
sameelarif merged 12 commits into
v4-spikefrom
sameelarif/stg-2749-cache-metadata-v4
Aug 3, 2026
Merged

[STG-2749] Surface cache metadata on act/observe/extract results#2543
sameelarif merged 12 commits into
v4-spikefrom
sameelarif/stg-2749-cache-metadata-v4

Conversation

@sameelarif

@sameelarif sameelarif commented Aug 1, 2026

Copy link
Copy Markdown
Member

why

Adds cache observability to result metadata. Previously a result only told you HIT or MISS but never why it missed, how close a key was to its threshold, or what a hit saved. The stateless cache API already returns all of it (hitCount, threshold, missReason); we parsed those fields and passed them straight to logger.debug.

what changed

Before

metadata: {
  actionId?: string;
  cacheStatus?: "HIT" | "MISS";
}

After

metadata: {
  actionId?: string;
  cache?: {
    status: "HIT" | "MISS";     // required, replaces metadata.cacheStatus
    count?: number;             // times this key has been seen
    threshold?: number;         // threshold in effect
    missReason?: string;        // misses only
    tokensSaved?: { inputTokens: number; outputTokens: number; totalTokens: number };  // hits only
  };
}

cache is present exactly when a lookup ran, so status being required means it always explains itself:

const { cache } = result.metadata;
if (!cache) console.log("caching not active");
else if (cache.status === "HIT") console.log(`cached, seen ${cache.count}×`);
else console.log(`live: ${cache.missReason}`);

Breaking: metadata.cacheStatusmetadata.cache.status

test plan


Summary by cubic

Adds a required metadata.cache object to all v4 act/observe/extract results to explain hits, misses, and token savings; reports DISABLED when no lookup ran. Completes Linear STG-2749 (v4 port of STG-2656) to improve debugging and cost visibility.

  • New Features

    • metadata.cache is required with status: "HIT" | "MISS" | "DISABLED", plus optional count, threshold, missReason, and tokensSaved (inputTokens, outputTokens, totalTokens default to 0); removed ageMs from results (still logged).
    • Populated across services; distinguishes local failures as read_failed and replay_failed. act now writes aggregate llmUsage so hits can report tokensSaved.
    • Updated protocol and SDKs (packages/protocol/*, packages/sdk-ts, packages/sdk-python, packages/sdk-go), v4 docs, and tests, including end‑to‑end coverage and cache‑client parsing for the tokensSaved wire shape.
  • Migration

    • Replace result.metadata.cacheStatus with result.metadata.cache.status (TS), cache_statuscache.status (Python), and the CacheStatus pointer with a required Cache struct and Cache.Status (Go).
    • No lookup is now result.metadata.cache.status === "DISABLED" (no null checks).
    • Read token savings from result.metadata.cache.tokensSaved.{inputTokens,outputTokens,totalTokens} on hits.

Written for commit 8ebf30a. Summary will update on new commits.

Review in cubic

cacheStatus alone could not explain itself: a miss never said why, a hit
never said how established the entry was, and MISS was stamped equally
for a cold cache, a failed cache read, and a cached value that could not
be replayed. The API already returns hitCount, threshold, ageMs, and
missReason on every lookup — cacheService parsed them and passed them
only to logger.debug.

Adds an optional cacheMetadata object to StagehandResultMetadata
carrying count, threshold, ageMs, missReason, and tokensSaved, populated
on both the hit and miss paths. Hit and miss metadata are built
separately so a result can never report both stories at once, and the
two local failure modes get their own reasons (read_failed,
replay_failed) instead of masquerading as a cold cache.

cacheStatus is unchanged, so this is additive for all three SDKs.
tokensSaved is plumbed through but stays undefined until the API's
stateless get forwards the savings it already computes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sameelarif
sameelarif requested a review from a team as a code owner August 1, 2026 01:06
@changeset-bot

changeset-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8ebf30a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages
Name Type
@browserbasehq/stagehand Minor
@browserbasehq/stagehand-python Minor
@browserbasehq/stagehand-go Minor
@browserbasehq/stagehand-server Minor
@browserbasehq/stagehand-evals Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

sameelarif and others added 2 commits July 31, 2026 18:08
cacheStatus and cacheMetadata said the same thing at two altitudes.
Collapse them into a single optional `cache` object with a required
`status`, so it is present exactly when a lookup ran and always explains
itself. Named `cache` rather than `cacheMetadata` to avoid
metadata.cacheMetadata stutter, and it mirrors the `cache` input option.

Drops ageMs: the entry's age is a server-side detail, still logged but
not worth a field on every result.

Updates the three SDK surfaces, examples, and v4 docs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…49-cache-metadata-v4

# Conflicts:
#	packages/sdk-go/internal/extensionassets/stagehand-extension.zip
#	packages/sdk-ts/examples/caching.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cubic analysis

Review completed against the latest diff

Linked issue analysis

Linked issue: STG-2749: Surface cache metadata (count, miss reason, tokens saved) on v4

Status Acceptance criteria Notes
Add a cacheMetadata object to StagehandResultMetadataSchema carrying count, missReason, ageMs, threshold, and tokensSaved Protocol and type schemas were added and wire formats updated; generated SDK models and docs were also updated to include cache metadata fields.
Populate cacheMetadata in cacheService.withCache on the hit path (including count, threshold, ageMs, tokensSaved when present) withCache maps server response fields into a hit metadata object and attaches it to result.metadata.cacheMetadata when present.
Populate cacheMetadata in cacheService.withCache on the miss path, including read-failure and onHit-threw fallbacks, with distinguishable miss reasons Miss handling builds cache metadata with missReason and includes different reasons for read failures and replay failures; miss metadata is attached when outcome is a MISS.
⚠️ Forward tokensSaved from the stateless cache GET response into CacheGetResponse so tokensSaved can be surfaced This PR adds client-side parsing and internal mapping for tokensSaved and maps it into cacheMetadata on hits, but the companion server/core change that actually forwards tokensSaved from handleStatelessCacheGet is not present here (the client accepts tokensSaved but the core change was noted in the issue as a separate required change).
Keep cacheStatus behavior unchanged (additive change) so existing clients/tests remain valid cacheStatus is still set to HIT/MISS as before and new cacheMetadata is additive.
Add tests for hit, miss, threshold miss, and cache-failure paths Unit tests were added that cover hits (including token savings), basic miss and persistence, threshold progress misses, read failures, and replay failures, plus a test for no lookup when caching disabled.

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/server/services/cacheService.ts
Comment thread packages/server/services/cacheService.ts
Comment thread packages/server/clients/cacheClient.ts
Comment thread packages/protocol/schemas.ts Outdated
Comment thread packages/protocol/schemas.ts Outdated
sameelarif and others added 2 commits August 3, 2026 12:10
Review feedback:
- CacheTokenSavings fields default to 0, matching StagehandResultUsage,
  so a hit always reports all three counts.
- metadata.cache is no longer optional. Making it required needs a
  representation for "no lookup ran", so CacheStatus gains DISABLED and
  services seed every result with it; withCache overwrites when it runs.
  Callers no longer need to null-check before reading status.
- act now reports its aggregate llmUsage on cache writes. It was the one
  primitive that sent none, so act hits could never compute tokensSaved.
- Adds cache-client tests for the tokensSaved wire shape and malformed
  payloads; cache-service tests mock past the parser, so a renamed field
  would previously only have surfaced in production.

Merge: keeps upstream's per-operation usage aggregate alongside cache
metadata, and rebuilds the Go-embedded extension, which was stale
against the merged server build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The service tests mock the cache client and the client test only parses
payloads, so nothing proved that a real API response becomes the right
metadata.cache. That seam is exactly where tokensSaved was being lost.

Drives the real CacheClient over real HTTP against a local server
speaking the API's captured response shapes, asserting the exact
metadata for each outcome: cold miss, hit with count/threshold/savings,
hit without recorded usage, threshold miss with progress, API failure
(read_failed), unusable cached value (replay_failed), and caching off
(DISABLED). Also pins that a write carries llmUsage and that a
per-request threshold reaches both get and set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/server/tests/cache-e2e.test.ts
Comment thread packages/server/tests/cache-e2e.test.ts Outdated
afterAll fired server.close() without awaiting it, so the suite could
finish while the fixture was still closing and leak the handle if a
request were still in flight. Drop live connections, then await the
close.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/server/tests/cache-e2e.test.ts Outdated
sameelarif and others added 2 commits August 3, 2026 13:29
The repo pins pnpm 11.10.0 (devEngines, and CI sets it explicitly "to
protect the lockfile"), but this branch was installed with pnpm 10,
which rewrote the lockfile into the older format and dropped its
configDependencies/packageManagerDependencies sections — 5k lines of
diff and a format CI would not have produced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sameelarif and others added 4 commits August 3, 2026 14:25
closeAllConnections() snapshots the tracked sockets, so calling it before
close() leaves a window where a connection accepted afterwards keeps
close() waiting. Close first, then drop — the order the SDK's integration
closeServer helper already uses.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nesting CacheMetadata inside StagehandResultMetadata pushed the literal
past the line limit; ruff format wraps it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…49-cache-metadata-v4

# Conflicts:
#	packages/docs/v4/reference/stagehand.mdx
Renaming the field from CacheMetadata to Cache left the struct literal
padded for the old, longer key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sameelarif
sameelarif merged commit 494fd0d into v4-spike Aug 3, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants