Using CDN like CloudFront, CloudFlare on App Directory breaks everything

[HamAndRock]

Hello this is my first PR, sorry about the mess :)

What?

With app directory being used for production app, we want to add CDN to cache HTML responses (yes we can leverage static rendering etc. + we are self hosting on AWS).

Hovewer we still ideally want to cache the HTML on a CDN, hovewer we ran into and issue where the same url for the page can return 2 payload depending on the content-type, with CloudFront and CloudFlare you can't cache on level of mime types. So we can't cache the RSC payload with x-component type and the HTML output with mime type HTML separately. This results into several errors when the RSC payload is replaced with HTML that was cached from CDN and router doesn't find HTML to match the spec of x-component and starts doing infinite loop of page refreshes.

Some screenshots of the cached payload and the errors statis that RSC is not matching what it should be and causing a page reloads infinitely.

This got cached on CDN first, returning just the RSC payload instead of HTML, after a while it started returning normal HTML but causing those RSC fetch fail errors.

Reproduce?

To reproduce create any app directory project with server components and add a CDN in front of it.

Why?

We want to leverage selfhosted next app director to use CDNs like CloudFront and CloudFlare.

How?

Add rscQuery query param to differ between RSC payload and HTML payload for CDN caching purposes. This is totally not the best solution but allowed use to have set of 2 different urls that we can cache on URL basis. I am not sure if adding this rscQuery prop doesn't break anything internally but we tested this on project that will be shipped soon and we ran into 0 issues, but again this is a "patch" level fix, the problem is more complicated probably.

[Fredkiss3]

Hello, there is an issue regarding this, one of the next core team answered that normally that next adds a Vary header by default to fix this, cloudfare & cloudfront don't respect them (which is weird, since it is a standard).

So i proposed a solution for cloudfare : #48569 (comment) which is basically what you are trying to do here, adding query params for each value in the Vary header. I think you could do the same with cloudfront (i don't really know) ?

[khuezy]

CloudFlare/Front not honoring Vary is weird... but to work around this issue, you can add the "rsc", "react-router-state-tree" cache control policy. That will differentiate between the /page and the x-component type.

[juanferreras]

cc @timneutkens would it be possible to get your opinion (or any other core member's) on this issue?

Next.js is doing things correctly by setting the Vary header, but if most CDNs users are leveraging (when not serving from Vercel) will not be compatible with this approach, it'd still block adoption (and/or harm perceived stability) for the app directory. Seeing users cite issues on Cloudflare (which might be proxying Vercel) and AWS Cloudfront, DigitalOcean and Firebase for self-hosted options.

Is there any downside towards a hybrid approach with query strings too? Many thanks!

(relevant tweet: https://twitter.com/matveydotgg/status/1655563317056774146)

[timneutkens]

We're going to solve this issue where cloudflare/firebase etc don't respect Vary soon, even when Next.js sets the right HTTP headers as per the spec. Some providers seem to only allow differential serving of requests on their enterprise plan. The idea right now is to change the RSC header to be one header instead of 3 and add a querystring hash of the router state tree.

The important context here is that we started with searchParams initially, but it turned out to not be the right fit as the url would get too long with the router state that the server needs, causing navigations to break. Additionally it would require additional parsing of the url to make e.g. middleware work correctly.

The wayback machine issue is different though, from what I'm seeing it seems their crawler is not correctly applying Vary, causing any fetch() and such to be cached. We've reached out to them via the available email address, haven't heard back on that yet.

[hirvesh]

@timneutkens - any help needed here to get this over the line? 🤔