Data Cache MISS at a rate much higher than expected

[florimondmanca]

Summary

Hello,

Our app uses App Router (migration from Pages Router in progress) and we use the Data Cache in our dynamic routes to cache GraphQL queries made with Apollo.

We've set the revalidation time with next.revalidate as part of Apollo fetchOptions to 24 hours.

Yet I am seeing cache MISS logs in Vercel for these queries multiple times an hour.

This is a much higher rate than expected. The hit rate in Vercel Observability is only 50%, we would expect 90%+. This increases backend load and our Vercel usage costs.

What could explain this? Why is the cache not honoring the revalidation rate? LRU limits? Something else?

Additional information

Some info on our app:

- Next 16, Turbopack, NOT cacheComponents (we tried but hit too many bugs), App Router (for pages where we see this behavior), deployed on Vercel.
- We have some marketing pages, but most of our app is a purchase funnel for a product among 20k+ products, identified by a slug. These pages a dynamic due to usage of cookies, searchParams, etc.
- I am seeing the non-honored cache behavior on the product detail page query in particular. It returns slow-changing data, hence why we want to cache it for 24h in order to avoid unnecessary load and compute. We are expecting one cache entry per product slug, so about 20k, which seems reasonable?
- Total daily Data Cache usage on Observability: 1.2M reads, 200k writes.

Example

No response

[icyJoseph]

I'd also try to pipe this through Vercel community, or support.

That being said, I wonder if in graphql, the variables used in the query are to high cardinality (higher than expected likely means not)? I need to re-read upon Vercel Data Cache... again

[icyJoseph]

Ah - bad habit of mine just skimming through the first time. Ok, so you do fill your caches, but they seemingly get evicted too quickly? I guess, https://vercel.com/docs/runtime-cache/data-cache#storage-and-eviction could be the thing right?

GraphQL queries tend to be a bit verbose 🤔

[florimondmanca]

Yes, this doc paragraph is why I mentioned "LRU limits?" in my post. But -- it says we can get info on cache eviction in Vercel Observability yet I only see cache usage info.

You are right that GraphQL query responsea can be quite verbose, and this could be counting towarss this "fixed storage limit".

At this point I started digging in the NextJS cache. I'm trying to find / confirm which code gets used:

A - General code location

In this case, am I right that the code being used lives under server/lib/incremental-cache?

B - LRU code

I am not sure where the LRU code being used lives, and most importantly where to find the "fixed storage limit" mentioned by the docs -- I suppose this takes into account max entry size, max entry count, max total cache size, etc.

C - Cache key computation

I think generateCacheKey() from incremental-cache/index.js is used?

I added some logs in the copy from my node_modules and it showed the contents used to generate the hashed cache key.

I see that a lot of info is being used, including request headers as well as the GraphQL query+variables+extensions.

That's a lot of potential for variability so I need to look into this.

Other experiments

I have experimented with a simple Map-based custom cacheHandler on local just to see calls to get() and set() being made, and log some stuff including the cache keys and options.

I also enabled NEXT_PRIVATE_DEBUG_CACHE=1 on local and just enabled this in a Preview branch so I will look into it.

[icyJoseph]

I still think this is something that the platform should help with, try https://community.vercel.com/ or https://vercel.com/docs/dashboard-features/support-center

I'll see what I can dig from my end.

[florimondmanca]

I filed a support ticket with Vercel, waiting for their input.

In the meantime I did more digging into the code

After activating NEXT_PRIVATE_DEBUG_CACHE=1, the build log shows this

IncrementalCache: using filesystem cache handler
FileSystemCache: using memory store for fetch cache

I am assuming this is used in production as well, i.e. the fetch() cache uses an in-memory store

Digging into the code for the incremental cache, AFAICT said memory cache is instantiated using this code:

next.js/packages/next/src/server/lib/incremental-cache/memory-cache.external.ts

Lines 7 to 34 in def0e3a

	export function getMemoryCache(maxMemoryCacheSize: number) {
	if (!memoryCache) {
	memoryCache = new LRUCache(maxMemoryCacheSize, function length({ value }) {
	if (!value) {
	return 25
	} else if (value.kind === CachedRouteKind.REDIRECT) {
	return JSON.stringify(value.props).length
	} else if (value.kind === CachedRouteKind.IMAGE) {
	throw new Error('invariant image should not be incremental-cache')
	} else if (value.kind === CachedRouteKind.FETCH) {
	return JSON.stringify(value.data || '').length
	} else if (value.kind === CachedRouteKind.APP_ROUTE) {
	return value.body.length
	}
	// rough estimate of size of cache value
	return (
	value.html.length +
	(JSON.stringify(
	value.kind === CachedRouteKind.APP_PAGE
	? value.rscData
	: value.pageData
	)?.length || 0)
	)
	})
	}
	return memoryCache
	}

The maxMemoryCacheSize option I traced being fed from the NextJS config, here

next.js/packages/next/src/server/next-server.ts

Lines 425 to 445 in def0e3a

	// incremental-cache is request specific
	// although can have shared caches in module scope
	// per-cache handler
	return new IncrementalCache({
	fs: this.getCacheFilesystem(),
	dev,
	requestHeaders,
	allowedRevalidateHeaderKeys:
	this.nextConfig.experimental.allowedRevalidateHeaderKeys,
	minimalMode: this.minimalMode,
	serverDistDir: this.serverDistDir,
	fetchCacheKeyPrefix: this.nextConfig.experimental.fetchCacheKeyPrefix,
	maxMemoryCacheSize: this.nextConfig.cacheMaxMemorySize,
	flushToDisk:
	!this.minimalMode && this.nextConfig.experimental.isrFlushToDisk,
	getPrerenderManifest: () => this.getPrerenderManifest(),
	CurCacheHandler: CacheHandler,
	})
	}

This means I should be able to configure it in my next.config.ts to set the fetch cache size required for our needs?

Not sure yet, as this is not documented. The only mention of maxCacheMemorySize in the docs is in Custom Next.js Cache Handler which shows we can set it to 0 in case of a custom handler.

I am not sure either what's the default value -- again, not documented. But I found hints in the code that it would be 52M roughly.

It appears here, and somewhere else as 50 * 1024 * 1024 but I can't trace that back.

next.js/crates/next-build-test/nextConfig.json

Line 14 in 99b4c96

"cacheMaxMemorySize": 52428800,

Then I wondered what is the unit of this cache size.

In the code for getMemoryCache() above I can see this:

next.js/packages/next/src/server/lib/incremental-cache/memory-cache.external.ts

Lines 16 to 17 in def0e3a

	} else if (value.kind === CachedRouteKind.FETCH) {
	return JSON.stringify(value.data || '').length

So the size of an entry in the fetch cache is defined as the number of characters of its JSON representation

I did some debug logging in our backend code to see what's the typical size (in these terms) of some GraphQL query responses we send to the frontend.

On one page which has 4000 variations (slugs), the query response is roughly 85k characters.

4000 x 85 000 = 340M

This is already 6x more than the default 50M max cache size

We cache these pages eagerly upon deploy. So by doing this, we are already exceeding the default cache size. If this default size is the actual size used by Vercel on prod for us right now, it means the fetch cache is constantly rotating, and only the really topmost requested pages benefit from the cache.

Does this analysis make sense to you?

Now I'm waiting to get confirmation from Vercel about the actual cache size used in prod, whether it can be configured as I saw, and what's the billing impact of this as well...

[florimondmanca]

I got the info I needed from Vercel support; they confirmed the LRU eviction hypothesis. We'll be considering options...

@icyJoseph Thanks a lot for your input!

[florimondmanca]

Update - Vercel confirmed we hit the Data Cache LRU limit and that as a result entries are being actively evicted.

Here's some info they shared that's not documented:

• Pro Plan: Data Cache limited to 128MB across all projects (this is us, and way too little)
• Enterprise Plan: Data Cache starting at 1GB per project, up to 15GB per project
• ISR remains unlimited in use (billed per usage)

I estimated our caching need to be roughly 10-15 GB, so it seems we'll need to move away from Vercel's Data Cache hosting.

It's sad that for the same Pro Plan in the past, we had a roughly working caching story under Pages Router -- pages were basically cached by ISR -- and App Router introduced limitations that forces us to rely on this severely limited solution.

• We cannot use ISR because it's full-static-pages only, and our pages are dynamic due to cookie-based frontend auth (as most product sites I guess?)
• And we won't be using Cache Components yet because we just did a 2-week attempt at migrating and encountered too many bugs to consider this stable at the moment.

As a result we will be investigating alternative cache storage solution, possibly relying on a Memcached instance already deployed for our backend needs.