Skip to content

feat: add runtime-controlled request-local caching#52

Merged
lan17 merged 6 commits into
mainfrom
agent/request-local-cache
Jul 17, 2026
Merged

feat: add runtime-controlled request-local caching#52
lan17 merged 6 commits into
mainfrom
agent/request-local-cache

Conversation

@lan17

@lan17 lan17 commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in request-local cache layer whose lifetime is the outermost enabled DialCache scope. Runtime policy can enable it per invocation with requestLocal: true, while existing local and Redis TTL/ramp behavior remains unchanged.

Closes #51#51

Architecture and design

Request lifetime

  • The outermost enable() creates a lightweight request holder.
  • Nested enable() and disable() calls reuse that holder while retaining async-scope-local enabled state.
  • Request-local value and in-flight maps are allocated together, lazily, only when an invocation resolves requestLocal: true.
  • When the outer callback settles, the holder closes and any maps are cleared. Detached async work inherited from that boundary becomes pass-through and cannot repopulate request-local state.

Runtime policy

  • DialCacheKeyConfig accepts optional requestLocal?: boolean; missing or false keeps request-local caching off.
  • Request-local policy has no TTL or ramp and is intentionally not part of the TTL/ramp-oriented CacheLayer enum.
  • Each enabled cached-function invocation fetches runtime configuration once and passes that snapshot through the selected chain.
  • A false decision bypasses request-local reads and writes without evicting an earlier scoped value, so a later true decision in the same scope can reuse it.

Lookup and coalescing

request-scoped single-flight
  -> request-local value
  -> instance-scoped single-flight (`scope="process"`)
  -> local LRU
  -> Redis
  -> fallback

Only active portions of the chain participate:

  • Request-local work coalesces before the request-local lookup.
  • After a request-local miss, active local/Redis work enters the instance-owned coalescer before the first shared layer.
  • With request-local off, active local/Redis work still coalesces within the same DialCache instance.
  • When every layer is off, calls remain independent pass-through executions.
  • Single-flight state is removed after fulfillment or rejection; only successful resolved values are memoized.
  • Cached undefined is preserved with membership-based lookup semantics.

Both value maps and flight maps continue to use the existing canonical DialCacheKey.urn. This deliberately avoids changing key encoding in this PR; the centralized typed-key work remains tracked by #14.

Invalidation, retention, and ownership

  • Targeted invalidation remains remote-only. invalidateRemote() does not evict or fence request-local or process-local values.
  • Request-local storage has no entry cap or eviction policy; the documented operating assumption is a short-lived scope with bounded key cardinality.
  • Request-local and process-local values are stored natively without cloning, freezing, or serializer round-tripping.
  • API and README guidance now make cached return values caller-immutable by contract and document layer-dependent reference identity.

The broader ownership test matrix remains tracked by #18.

API and observability

  • Adds DialCacheKeyConfig.requestLocal.
  • Uses the fixed metric-layer value request_local without adding request-local policy to CacheLayer or expanding the root value exports.
  • Adds required, bounded coalescing scope values request_local | process to custom adapter labels and exports the CoalescingScope type.
  • Defines one coalescing metric contract: dialcache_coalesced_counter{use_case,key_type,scope}.
  • Adds request-local request, miss, and get observations with layer="request_local".

Metric evolution and config observability follow-ups remain tracked by:

Documentation and maintainer tooling

  • Documents the minimal request-local-only configuration first, then separates request-local, process-local, and Redis behavior.
  • Covers HTTP request-boundary usage, nested scope behavior, lazy allocation, retention, invalidation, coalescing, and immutable value ownership.
  • Adds the repository-only pnpm benchmark:request-local maintainer command for sequential request-local hits plus request- and instance-scoped fan-out.
  • Keeps the published package limited to runtime artifacts; package-consumer checks verify the public ESM/CJS/type surface.

Validation

Run with Node v22.22.0:

  • corepack pnpm check
    • 130 unit tests passed
    • coverage: 97.63% statements, 91.20% branches, 98.03% functions, 97.72% lines
    • ESM/CJS declarations and packed-package consumer checks passed
  • corepack pnpm test:integration
    • 46 Redis integration tests passed
  • corepack pnpm benchmark:request-local
    • 50,000 sequential request-local hits: one fallback
    • 1,000 request-local coalesced calls: one fallback
    • 1,000 same-instance scope="process" coalesced calls: one fallback
  • Packed ESM/CJS consumers compile the optional structural config shape plus the required scoped metric labels and exported CoalescingScope type.
  • Regression coverage includes active request-local work settling after a rejected outer scope, per-instance scope="process" isolation, continuation through the remaining active layer after ramp-resolution failures, and fail-open handling when the coalesced() metrics hook throws.
  • Fresh lifecycle and cache-chain reviews found no implementation defect; the API/docs review found one instance-boundary wording issue, corrected in the README and tracker text.
  • git diff --check

@lan17
lan17 marked this pull request as ready for review July 17, 2026 05:25
@lan17
lan17 merged commit 53457d0 into main Jul 17, 2026
4 checks passed
@lan17
lan17 deleted the agent/request-local-cache branch July 17, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a runtime-controlled request-local cache layer

1 participant