feat: implement LRU cache with invocation ID scoping for minimal mode response cache #88509
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wyattjoh
force-pushed
the
wyattjoh/response-cache-lru
branch
from
6bbc429 to
d9faef8
Compare
Member
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Collaborator
Tests Passed |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Collaborator
Stats from current PR🔴 1 regression
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles: **432 kB** → **432 kB** ✅ -53 B82 files with content-based hashes (individual files not comparable between builds) Server Middleware
Build DetailsBuild Manifests
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📝 Changed Files (25 files)Files with changes:
View diffsapp-page-exp..ntime.dev.jsfailed to diffapp-page-exp..time.prod.jsfailed to diffapp-page-tur..ntime.dev.jsfailed to diffapp-page-tur..time.prod.jsfailed to diffapp-page-tur..ntime.dev.jsfailed to diffapp-page-tur..time.prod.jsDiff too large to display app-page.runtime.dev.jsfailed to diffapp-page.runtime.prod.jsDiff too large to display app-route-ex..ntime.dev.jsDiff too large to display app-route-ex..time.prod.jsDiff too large to display app-route-tu..ntime.dev.jsDiff too large to display app-route-tu..time.prod.jsDiff too large to display app-route-tu..ntime.dev.jsDiff too large to display app-route-tu..time.prod.jsDiff too large to display app-route.runtime.dev.jsDiff too large to display app-route.ru..time.prod.jsDiff too large to display pages-api-tu..ntime.dev.jsDiff too large to display pages-api-tu..time.prod.jsDiff too large to display pages-api.runtime.dev.jsDiff too large to display pages-api.ru..time.prod.jsDiff too large to display pages-turbo...ntime.dev.jsDiff too large to display pages-turbo...time.prod.jsDiff too large to display pages.runtime.dev.jsDiff too large to display pages.runtime.prod.jsDiff too large to display server.runtime.prod.jsDiff too large to display |
wyattjoh
force-pushed
the
wyattjoh/response-cache-lru
branch
from
d9faef8 to
79e5ce4
Compare
ijjk
reviewed
Jan 15, 2026
ijjk
reviewed
Jan 15, 2026
wyattjoh
force-pushed
the
wyattjoh/response-cache-lru
branch
3 times, most recently
from
0a9f405 to
1bca000
Compare
wyattjoh
force-pushed
the
wyattjoh/response-cache-lru
branch
from
ed2b3fc to
92c785d
Compare
Merging this PR will not alter performance
Comparing 
Footnotes
|
wyattjoh
force-pushed
the
wyattjoh/response-cache-lru
branch
2 times, most recently
from
b357e62 to
e09d56b
Compare
ztanner
reviewed
Jan 20, 2026
ztanner
reviewed
Jan 20, 2026
wyattjoh
force-pushed
the
wyattjoh/response-cache-lru
branch
from
a3acfbc to
cecc2da
Compare
ztanner
approved these changes
Jan 20, 2026
wyattjoh
changed the title
ztanner
reviewed
Jan 21, 2026
ztanner
reviewed
Jan 21, 2026
ijjk
reviewed
Jan 22, 2026
ijjk
reviewed
Jan 22, 2026
wyattjoh
changed the title
Replace single-item cache with 50-entry LRU cache in ResponseCache for minimal mode. Uses actual revalidate TTL instead of hardcoded 1 second, and properly handles expired entries.
- Add `maxResponseCacheSize` experimental config option to control LRU cache size - Scope in-memory cache entries by `x-vercel-id` header (invocationID) to ensure request isolation in minimal mode - Add `infra` test utilities for testing cache deduplication with grouped requests - Update standalone mode tests to use new infra utilities for proper test isolation
With invocation ID scoping, TTL-based expiration is unnecessary since entries expiring mid-revalidate makes no sense. The LRU cache now handles eviction purely based on capacity.
Add logging when cache entries are evicted and the same invocation ID is used again, recommending users increase maxResponseCacheSize. - Add onEvict callback parameter to LRU cache - Track evicted invocation IDs in ResponseCache (bounded to 100) - Warn once per invocation when eviction detected - Add unit tests for onEvict callback
Add time-based expiration (10s) as fallback when invocationID is undefined. This handles non-Vercel providers that don't send x-vercel-id header yet. Changes: - Add expiresAt field to cache entries for TTL-based expiration - Use invocationID scoping on Vercel, TTL fallback on other providers - Increase default maxResponseCacheSize from 5 to 10 - Add JSDoc clarifying remove() doesn't trigger onEvict callback
- Rename header from vendor-specific x-vercel-id to generic x-invocation-id - Extract FALLBACK_TTL_MS constant with documentation - Increase default LRU cache size from 5 to 10 - Update all tests to use new header name
- Add peek() method to LRU cache for read-only checks without affecting eviction order - Set TTL expiration for ALL cache entries (not just those without invocationID) - Schedule automatic eviction via setTimeout after TTL elapses to reduce memory pressure - Add experimental.responseCacheTTL config option (default: 10000ms) - Remove Vercel-specific language from code comments
Replace setTimeout-based proactive eviction with lazy LRU eviction. Memory is already bounded by maxSize, so timer-based cleanup was redundant and could accumulate thousands of pending timers in high-throughput scenarios. TTL-based cache hit validation is preserved via expiresAt checks.
The peek() method was only used by the timer-based proactive eviction which was removed in the previous commit.
- Fix typo in request-meta.ts ("it's" → "its")
- Expand FIFO eviction comment with detailed rationale
Ensures user configuration (maxResponseCacheSize, responseCacheTTL) is always respected by requiring nextConfig at all call sites.
Removes "(standalone deployments)" from config comments since minimal mode is not equivalent to standalone deployments.
Replace single-level LRU cache with a two-level structure: - Outer level: keyed by pathname (configurable via minimalModeResponseCacheMaxPaths) - Inner level: keyed by invocationID (configurable via minimalModeResponseCacheMaxInvocations) This allows multiple concurrent invocations to cache the same pathname without overwriting each other's entries, improving cache hit rates during parallel revalidation scenarios. Config changes: - Rename maxResponseCacheSize to minimalModeResponseCacheMaxPaths - Add minimalModeResponseCacheMaxInvocations (default: 5) Also extract helper methods to reduce code duplication: - trackEvictedInvocation(): handles FIFO-bounded eviction tracking - removeInvocationEntry(): removes invocation and cleans up empty paths
Move hardcoded default values to named constants at module scope: - DEFAULT_MAX_PATHS (30) - DEFAULT_MAX_INVOCATIONS_PER_PATH (5) Consistent with existing DEFAULT_TTL_MS pattern.
Replace experimental config options with environment variables: - NEXT_PRIVATE_RESPONSE_CACHE_TTL (default: 10000) - NEXT_PRIVATE_RESPONSE_CACHE_MAX_PATHS (default: 30) - NEXT_PRIVATE_RESPONSE_CACHE_MAX_INVOCATIONS (default: 5) This simplifies the API by removing the nextConfig parameter from getResponseCache() and reading configuration at module load time.
Replace the two-level LRU cache (pathname → invocationID) with a single LRU cache using compound keys (pathname + null byte + invocationID). This reduces code complexity while maintaining the same functionality: - ~50 fewer lines of code - Single NEXT_PRIVATE_RESPONSE_CACHE_MAX_SIZE env var (default 150) - Simpler eviction tracking via compound key parsing
…lper Replace the `infra` namespace in test utilities with a more explicit `withInvocationId()` helper per PR feedback. Tests should be very explicit about when they're passing an x-invocation-id header. - Add `withInvocationId(opts?)` helper that creates RequestInit with unique x-invocation-id header - Migrate 184 usages across 5 test files to use the explicit helper - For grouped requests (cache deduplication), use shared opts variable - Remove createInfraGroup() and infra namespace exports
wyattjoh
force-pushed
the
wyattjoh/response-cache-lru
branch
from
49df06b to
b07a6d3
Compare
ztanner
approved these changes
Jan 22, 2026
- Add parsePositiveInt() helper to validate environment variables, falling back to defaults for invalid values (NaN, non-positive) - Change TTL_SENTINEL to use null byte prefix, preventing collision with real invocation IDs since HTTP headers cannot contain null bytes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements an LRU cache with compound keys for the minimal mode response cache to improve cache hit rates during parallel revalidation scenarios.
Problem: The previous single-entry cache (
previousCacheItem) keyed by pathname caused cache collisions when multiple concurrent invocations (e.g., during ISR revalidation) accessed the same pathname. Each invocation would overwrite the previous entry, leading to cache misses and redundant work.Solution: An LRU cache using compound keys (
pathname + invocationID) that allows multiple invocations to cache entries for the same pathname independently:Cache Key Strategy
x-invocation-idheader: Entries are keyed by invocation ID for exact-match lookups (always a cache hit if the entry exists)__ttl__sentinel key and validate via expiration timestampConfiguration via Environment Variables
Cache sizing can be tuned via environment variables (using
NEXT_PRIVATE_*prefix for infrastructure-level settings):NEXT_PRIVATE_RESPONSE_CACHE_MAX_SIZENEXT_PRIVATE_RESPONSE_CACHE_TTLLRU Cache Enhancement
Added an optional
onEvictcallback toLRUCachethat fires when entries are evicted due to capacity limits. This enables tracking evicted invocation IDs for warning detection without introducing timer-based cleanup.Eviction Warnings
When a cache entry is evicted and later accessed by the same invocation, a warning is logged suggesting to increase
NEXT_PRIVATE_RESPONSE_CACHE_MAX_SIZE. This helps developers tune cache sizes for their workload.Additional Changes
x-vercel-idtox-invocation-idfor claritywithInvocationId()test helper for cache testingTest Plan
LRUCacheincludingonEvictcallback behaviorwithInvocationId()helper