RFC: Incremental Static Regeneration

[timneutkens]

Update: This has been released. See the docs for more information. The RFC will remain for historical context.

Summary

Fully automatic re-rendering of statically exported pages without a full rebuild.

With regards to static site generators this feature has been referred to in multiple ways, most commonly "Incremental builds" or "Incremental rebuilds".

This RFC exclusively discusses API additions. All new functionality is completely backwards compatible and can be incrementally adopted. This RFC introduces no deprecations.

Background

Next.js recently introduced support for next-gen Static Generation (SSG) in v9.3.

Generating static pages in Next.js using getStaticProps and getStaticPaths provides numerous upsides:

• Faster Websites: Generated files can be served from a global CDN close to your users
• Reduced Costs: Request are not rendered on-demand for every user

However static generation as it exists today has one downside that makes it hard to adopt for larger websites or applications:

Every time something changes in the database a full rebuild is required in order to see the change in one single page.

An example to illustrate this is when you make a change to a blogpost in a headless CMS, you could be only updating the title of one blog item. After you make the change the application would require a full rebuild of all pages, just to make that one change visible.

In order to make static sites scale we have to regenerate pages without needing a full build of the application.

What if we could generate a new version of the page on-demand when a new request comes in, but without having the user wait for the re-rendering?

Effectively re-rendering the page in the background while the previous version of the page keeps being served.

The background re-rendering could be triggered based on a timeout, similar to the way that Cache-Control's maxage works, when the timeout is exceeded a background re-render is triggered when a request comes in.

This would look something like:

• The page defines what timeout is reasonable, this could be as low as 1 second
• When a new request comes in the statically generated page is served
• When a new request comes in and the defined timeout is exceeded:
  • The statically generated page is served
  • Next.js generates a new version of the page in the background and updates the static file for future requests
• When a new request comes in the updated static file is served

This model is pretty ideal for static generation:

• Works in the background, users always get a static response
• Pages get statically re-generated, including all changes from the CMS
• It's fully automatic, no need to trigger a full rebuild

This allows for true incremental rendering of static pages in Next.js.

After this RFC for incremental static generation is implemented we can extend it to do force revalidation, which would be useful for CMS providers that want to get the content updated as quickly as possible.

Implementation

Next.js 9.3 introduced getStaticProps, in getStaticProps you'll be able to return a revalidate property in order to opt-in to automatically generating in the background when a request comes in:

// pages/blog/[slug].js
export async function getStaticProps({ params }) {
  const post = await getBlogPostBySlug(params.slug);
  return {
    // Set the timeout for generating to 1 second
    // This timeout could be longer depending on how often data changes
    revalidate: 1,
    props: {
      post,
    },
  };
}

Keep in mind that the timeout defined is a timeout for when an incoming request will re-render in the background. Meaning that the first request after the timeout period will serve the previously rendered page and then the page is regenerated in the background.

When you opt-in to revalidate you get:

• Fully static rendering
• Automatic re-generation in the background when a request comes in

Co-authored with @rauchg @Timer
Special thanks to @juancampa @dav-is

[Timer]

So excited for the future!

[jpsc]

You would still need a full rebuild if you change something shared with all pages eg header, correct?

[oncet]

Looking forward to this. Same case as mentioned above, I have a product page and showing an outdated price could be problematic. Right now my alternative was to fetch the price on client doing an AJAX request.

[anbeangry]

Hey guys, have it already solved ?

[derweili]

@timneutkens a force regenerate through an API would be really helpful. I'm currently using SSR with Redis cache instead as a workaround and and API route to clear the cache.

Replacing Redis with Static Generation and a force regenerate API would be great.

[thenglong]

+1. I saw a lot of use cases for this.

[JonatanLindstrom]

For anyone who missed it, this "forced regeneration" has now been released as "on-demand ISR".
See comment from from @leerob

[kevinwolfcr]

This looks awesome, a great proposal 🚀

As you stated on the upsides:

Reduced Costs: No server-resources are needed to render every request on-demand

1. If a specific page is requested several times throughout the day, doesn't this mean a lot of API/DB calls to check if the content has changed?
2. Will it work for getStaticPaths? (i.e. adding or deleting a blogpost from a CMS)

[jaybytez]

@Timer, great work on the new features. Any update on the goal for an on-demand (triggered via API route). We are working on webhooks for clearing our CMS & CDN caches and a way to force validation without setting the time limit super low would be great.

[kodehode]

We'll be exploring on-demand (triggered via API route) revalidation instead of interval-based in the future.

This would be greatly appreciated. We already have a case now where we opted to go the CloudFlare route because this feature is missing.

Also: when you look into this - will you look into how to do revalidation across multiple nodes/dynos? We host a site on heroku today running on 4 dynos - so the call should ideally trigger revalidation on all nodes at the same time.

[ats1999]

Great Idea!

[FSaldanha]

Related: #22384 and #22699 (see PR author's comment below)

[kristijorgji]

@tim-hanssen Any update on event based cache invalidation ? cli next invalidate url or admin api in next, to call POST domain.com/nextadmin/invalidate/url with some admin token or other auth.

[borekb]

Great to see another RFC in this area. A few specific points of feedback:

1: It took me a while to realize that this would happen on the server (or a serverless platform like Now) at runtime. The word static is commonly associated with a build-time process, producing something that can be uploaded to a "dumb" CDN.

I sort of realize now that you're talking about the end user experience – they would always get a static HTML – but this RFC basically proposes a behavior that's similar to SSR + nginx microcache (please correct me if I'm wrong) and that's not commonly referred to as SSG.

2: With these micro-caching strategies, it's often problematic to set the "right" timeout. In fact, there often isn't any one value. That's why most CMS's don't send aggressive cache-control headers and prefer to feel "slow" rather than outdated. I guess it's also why the example uses 1 second 😄.

You could argue that if you know your site, you can set this number sensibly but in my experience, it's always a tough choice.

3: In the RFC, it might be worth pointing out what this background re-rendering means for the backend uptime guarantees. With SSG, I can turn the headless CMS off after I've generated the site; here, I don't.

I assume that if the response from the backend is an error, the previously rendered HTML is still returned. This could also lead to situations where /post/my-blog-post already has the new title but /category/my-category/page/2 still displays the old value (backend was unreliable; I used a different timeout for that page, etc.)

[javiercr]

Thank you @Timer! Would it be possible to have some more detail on this? 🙏

This feature would be collaborated on in conjunction with Google (to ensure it doesn't cause negative SEO).

Would that collaboration means that Googlebot will treat Next.js different from other frameworks or just that Google has changed their definition or guidelines for Cloaking?

because their search engine is totally fine with blocking fallback mode

Does this mean that they're ok with getting Googlebot "blocked" (waiting for the SSR with the whole content) while the users get a different experience (skeleton first, then content fetched and rendered client-side)?

[balazsorban44]

The SSR way is how most of the sites are working nowadays I would guess, it is not a different type of "blocking" from the bot's perspective, than a slightly slower page. It is blocking in that manner that the user is blocked from real (or fake as in skeleton) content for a little longer. You see a blank screen a bit longer. The added "blocking" fallback mode just mimics that, so Googlebot will treat it as before. The interesting part is how Googlebot can tell when the fallback is actually just a skeleton page without real content? Now that might have to do something with Google treating Next.js differently, or they worked out some new standards in collaboration.

[valse]

Am I the only one who isn't worried about Googlebot? We had already published a SSG website with a lot of pages statically rendered thanks to the fallback support... No issues about SEO performance: the page speed is decreased a lot from the previous SSR website version because I think that the bot scan mainly the static page instead of the fallback one 😉

[balazsorban44]

As @Timer pointed out, the reason for a blocking fallback is not necessarily to make Google happier, but other crawlers like Facebook or Twitter. That was my only concern as well.

[javiercr]

The problem with SEO is that it's a very obscure topic and Google often doesn't provide enough clarity, so we all tend to play along with the few things we do know for sure.

What we know for sure is that serving Googlebot with content-rich HTML as fast as possible works (and that's why people like Static Site Generation).

However, IMHO serving content-rich HTML (thanks to blocking SSR) to Googlebot, and a skeleton (+ client side data fetching and rendering) to users, falls under the dated but official definitions of cloaking.

How does Google checks for cloaking? That's a mystery, therefore it's hard to tell whether they would notice the difference. If they did test this by fetching as Googlebot with JS disabled and then as "user" (no Googlebot user-agent) with JS enabled, the final result (after client side rendering) would be the same, and they wouldn't be able to tell the difference. However if they tested the user experience just by changing the user-agent but without enabling JS, they would see just the skeleton, and they may think the Googlebot version is cloaked to make the site look like if it had content that doesn't have.

Don't take me wrong, I do think the "smart mode" makes a lot of sense, and it's the best experience possible for everyone, but we need to be confident Google is ok with that too. Any further details would be more than appreciated!

[janhoogeveen]

First up, I really like this RFC. You folk already had me interested with getStaticProps and getStaticPaths, but this is next-level exciting! 🎉

My initial response:

1. How are revalidated pages stored? Like technically? At first I thought this was just a simple memory cache. The mention of a CDN now leads me to believe you are actually saving static files, in exactly the same way as when I'd run next build? That'd be really cool!

2. Would this work well for a situation where you have thousands or tens of thousands (public!) variations of a page? For example, could I simply not define getStaticPaths and let the built happen on-demand once deployed to a server? Any immediate downsides to that approach? I expect a small delay for the first visitor, but besides that?

3. This is not a question, merely a small suggestion... It might be nice to explain how this differs from Serverless Pre-Rendering. Both ISG and SPR have their unique use cases, but they're hard to differentiate if you're not already really familiar with both concepts.

[Timer]

@kevinwolfdev @fabb it is best practice to always run a proxy in front of Next.js when self hosting (for example, Nginx).

Nginx, when self hosting, is normally responsible for Gzipping responses from next start instance(s) and caching static assets in memory (/_next/static).

With the introduction of iSSG, you'd now want to configure your reverse proxy to cache your iSSG-powered pages by respecting the headers returned by next start (iSSG pages have s-maxage in their response). This cache would be shared across all of your reverse proxy (Nginx) instances, thus fixing the "divergence" of incremental rebuilds across instances. You'd want to have upstream-affinity keyed on the request path and load balanced appropriately (if not using a shared volume for the dist directory).

To deal with upstream failover, you could either tolerate the potentially stale read, or setup your next start instances to share a writable volume so they all read from the same copies.

Solving all of this complexity is what ZEIT Now is designed to do for you! But we respect the fact that you may want to self-host this.

---

Note: the quick and dirty way to get this working when self-hosting without any reverse proxies with shared caches would be to share the disk volume of all your Next.js server instances (K8S makes this easy). We don't suggest a proxyless-setup though, even without iSSG.

[janus-reith]

@Timer So basically, I would still have multiple nextjs instances, not for actually handling many requests to static pages as they would only reach the cache/proxy, but just in case many fallback/revalidated routes would be requested at the same time, or for handling api routes which I, depending on their purpose, might omit from caching.

So as the proxy would cache based on s-maxage, there would always just be one request to one of the nextjs instances, and therefore no instances having to build the same page at time.
Sorry if I'm mostly just repeating, I wanted to make sure I got things right.

In the case I wouldn't use api routes our just a fixed amount of possible api calls which could be cached, and also no fallback routes and only a fixed amount of pages that would possibly need to be revalidated at the same time, there would be no need to scale nextjs at all and only scale at the proxy/cache level I guess?

And in a more sophisticated scenario I would generally split these concerns, and have something that serves static html + js asap over cdn, with some fallback to a serverless function if a page doesn't exist/isn't valid anymore to create it, and serverless functions for api routes - Which probably is comparable what Zeit does for you, right?

[kristojorg]

@janus-reith I was also curious how this would work with serverless functions, or rather how it is very different from deploying Next.js as serverless functions behind a CDN cache? It seems more complicated than that route because the instances would have to share volumes, whereas with serverless, the pages are not responsible for caching, they just generate the page when requested, and then the CDN (possibly with a max-age or programmatic invalidation) stores the static files.

I guess I'm struggling to see how this is better than that approach?

[janus-reith]

@kristojorg One difference I see is, that you would have an immediate response from disk with the fallback on first page loading instead of awaiting the full page to be build first, although that behaviour is what some actually don't want to have and prefer the increased response time instead #11552 (reply in thread). And certainly, it benefits those who don't use a CDN or Cache in front of nextjs.
Regarding the shared volumes, that seems to be true if you don't do one of the two options @Timer mentioned, as although the cache would usually just hit one instance once during the valid time, it could be a different instance for the next validation period and you could get the fallback page again.
(Or maybe not, if the CDN would get the stale-while-revalidate instruction and present the original page while still triggering a new build? - I only have little knowledge of that topic, but in that case it doesn't seem necessary to me to share volumes or to take measures regarding upstream-affinity).

I also realized that even with a programmatic approach to trigger revalidate, I would still need to tell the CDN to actually revalidate, so this really seems to mostly benefit solutions like Now, where these things are connected.

If the serverless nextjs implementation catches up on these features, that might also give an added value here of having things under one hood if using something like Cloudfront together with Lambdas.
Regarding that I stated some of my - possibly incorrect- considerations here: serverless-nextjs/serverless-next.js#277 (comment)

[fabb]

With the introduction of iSSG, you'd now want to configure your reverse proxy to cache your iSSG-powered pages by respecting the headers returned by next start (iSSG pages have s-maxage in their response).

@Timer wouldn't that cause pages to be stale for at least 2x the s-maxage time? 1st time for the reverse proxy cache, and then for the 2nd time the next request lands on the Next.js server which returns the stale response (which is the same as the one in the proxy cache when using upstream-affinity keyed on the request path, or a shared volume) and triggers a rebuild - the stale response will be cached in the reverse proxy cache again for s-maxage. Only the next request after that time will fetch the regenerated page.

[wceolin]

Great proposal, thanks for this!

I'd love to see an API for event-driven changes. @Timer already mentioned it's on your plans, so I'll just share my current use case and why it would be helpful.

Currently, I'm using Next.js with a Firebase backend to build a collaborative platform for learning where users can edit any page (similar to how Wikipedia works). This means a page can change multiple times a day. This RFC would allow me to use getStaticProps and have faster loading times. However, it wouldn't allow me to save much money on Firebase read requests because I don't know how often that page will be updated. So, I'd have to set revalidate to a low number.

However, I could save lots of money if we'd have an event-driven API because I could create a Firebase Cloud Function for listening to post changes, for example, and send a request to update that post's page only.

Anyway, I'm glad to see the direction you're going. I've been working with Next.js for less than a year but it has been a great experience so far. Thanks for your hard work!

[jthegedus]

You may find this Firebase talk interesting - https://youtu.be/7_2CJs_VZk4
It covers content-change rendering with Firebase.

[robertwbradford]

An event-driven API would be extremely useful—or additional option on the current build script. We are looking more into static page generation but could potentially have 1000s of pages in various languages. Being able to programmatically "push" regeneration of an individual page when its content is changed would be ideal (e.g., fixing a typo in the CMS).

Or, perhaps, if a whole subdirectory of a site had a common element change, it would be nice to tell a collection of pages to regenerate—maybe through a regex/glob/list of path(s).

[maraisr]

I have this feature today. We run nextjs in a docker container in Fargate. We have a CDN that fronts all of that. Each request hits our CDN, then our origin (Nextjs). If the page exists in the CDN, the origin doesn't get hit, bit it doesn't then nextjs serves the response.

When "our database updates", we just proc a cdn cache invalidation to that changed route, and depended routes for that matter.

The only thing I see this new pattern solve, is being able to de-bounce failed responses, with previous successful responses.

We serve close to 300k distinct pages out of nextjs, if we flick on SSG - would we need disk space to house all of those html docs?

[janus-reith]

Although this may vary heavily, for reference my pages use Material UI and some other libraries(i.e there are not the smallest), and the generated revaliated fallback pages have a size of ~100 kb each. Add to that the size of the accompanying json file with the generated pageProps, in my case these are around 10kb each as they contain a lot of product data.
Nextjs stores only the latest copy, so with 300k of these pages I would probably be at around 33 Gigabytes. In my case Im rendering per locale, so for two locales I offer on the page that would be 66 GB, just in case i18n is a concern.

[radosslav]

When "our database updates", we just proc a cdn cache invalidation to that changed route, and depended routes for that matter.

This approach work on individual pages, but what about some listing pages and search results?

If someone changes the title of the article/product, you purge only this individual URL, but everywhere else CDN continues to returns the old one.

[aralroca]

Great proposal!

I have a question. In our e-commerce we are using SSR returning different content depending on the delivery country (stored on a cookie), but the page route can be the same (bots always see the default delivery country because they don't change the country).

Is there any plan to support this? Or the only way is to add an extra slug with the delivery country...?

[aralroca]

@vgrafe We published an i18n lib SSG-friendly with next.js (it also works for fully static site): https://github.com/vinissimus/next-translate

The solution has an extra build step with a CLI, as a temporal workaround meanwhile there is not a better solution...

[janus-reith]

@aralroca I‘d argue there is one, by using dynamic routing with some locale path segment and getStaticProps + getStaticPaths.

No added build step, but therefore some added complexity when routing between pages (Can be mitigated by a wrapped Link in a custom component to add the locale segment) and some added verbosity to fetch the translation namespaces per page.
A big advantage with that approach is, that you can fetch translations from an external source during fallback or revalidate, which adds some new possibilities.

Might be a matter of personal preference and project requierements.

Related discussion: #10651
Some outdated example I need to update soon: https://codesandbox.io/s/nextjs-i18n-staticprops-92ebn

[vgrafe]

I see. Thanks for the leads! @janus-reith , your solution seems similar to the one I linked above. I'd go all in if only the [lang] was an optional path segment... I know it is a bit extreme of a requirement but I'd like not to break our webapp's existing ref/promo URLs.

I will definitely look into it tho. The github thread you linked is great too. Thanks!

[janus-reith]

@vgrafe Yes that's also where I took inspiration from, although that was not build with static generation in mind and used getInitialProps.

Regarding your existing URLs: #9081
You can use rewrites and redirects to keep them working, or also make sure all existing paths still work: I did that with the help of a catchall api route which detects the language from browser headers, then takes the subpath and redirects to the according new page.
e.g. that would be /somepage =rewrite=> /api/detectLanguage/somepage =redirect=> /en/somepage.
But I will better stop at this point to not go more offtopic.

[vgrafe]

This is so helpful. Thank you @janus-reith 💯

[Icehunter]

How would this work if the site assets are built with a CDN prefix; in addition to having multiple copies of the same container/build running in a ECS cluster, around 200 of them at the same time?

[timneutkens]

You'll want to read this thread #11552 (reply in thread)

[tpreusse]

Awesome proposal and I love how fast you are moving forward with static generation options. Combined with the fallback option I'm starting to consider using it.

We're running republik.ch, an Swiss German online magazine, on next.js starting with v2 and running v9.3.4 right now. Currently we're still 100% server-side rendering—using apollo in getInitialProps. I'd like to move to a more hybrid approach and start useing the static features of next.

While a bit off topic, here is what currently is still blocking me:

• being able to return proper 404s with fallback: true
• flexible segmenting into two or more user groups with different caches
  • public / no cookie, cache interval 5 minutes, html with pay notes
  • signed in, cache interval 1 minute, html without pay notes
  • editors, live mode—no cache

I considered abusing preview mode for the «signed in» segment but that felt wrong. Maybe preview mode could be made more flexible—allowing a custom function to determine whether to live render or not. For us live rendering for signed in users would be an option because we've already build an html partial caching solution.

The non-plus-ultra solution for us, and maybe other hybrid users, would be to have a custom segmenting function that determines which cache and caching strategy should be used for a request. Maybe this function could also solve the 404 case: use live mode if not part of the current static path list.

[JonCognioDigital]

This is the bit I can't get my head around, when using fallback:true a status 200 is returned every time. 301s, 302s and 404s all not possible, so not great for search engines unless I've missed something.

[johannessteu]

@tpreusse Did you already found a solution for the 404 issue?

[Zauberfisch]

bump, any update on this?
Perhaps we should raise this as a separate issue, given that this feature has already been implemented?

I agree, not being able to return a non-200 http header is problematic.
I see how it's technically impossible to do with fallback: true. But with fallback: "blocking", it should be possible to do.

[timneutkens]

Have a look at #17755

[Zauberfisch]

@timneutkens ah, excellent. hot of the press I see. I'll keep my eye on that then. Thank you

[valse]

One of the big feature of a SSG website is the ability to put it on a CDN: incremental build for me is like a partial CDN purge and about it for my static contents I'm using Fastly and its Surrogate Keys feature: you can tag a content with one or many tags as you want and then purge it by tag and not a full purge of the cached contents.

In this direction I think rebuild in background only the pages tagged by a key could be the right approach (calling an API with one or more keys).

[Vadorequest]

Smart and yet simple way to go about relationship between pages. It's also very much simpler to understand from a human point-of-view about how cache invalidation would work.

@Timer Are you considering such approach? I read above you were already working/designing cache invalidation by API, what do you think of this implementation?

[radosslav]

you can tag a content with one or many tags as you want and then purge it by tag

Yes, but Fastly have a 1K API calls per hour limit - for some busy sites like e-commerce or classifieds, users can hit these limits very fast - we always have an option to put all of them on any kind of Queue service for the subsequent execution, but this is definitely not a solution when you need guaranteed consistency.

[lucasconstantino]

Just released a package to add cache-tags/surrogate-keys based ISR: next-cache-tags. Still in early stages, but I hope it can push for some core-integrated solutions to come.

[slorber]

If you have a cluster of many NextJS instances running at the same time, and node1 receives a request and regenerates a static page, wouldn't node2 (not hit by any request currently) be out of sync?
Given enough time, all nodes might start to become out of sync, and have different revalidation timestamps no?

I guess it could be mitigated by having a caching layer on top, and ensuring that if node1 revalidates, then the cache will not hit node2 before the new node1 static content expires.
But will it always be the case in practice? Will this solution work if we just put a simple round-robin load-balancer on top of the cluster?
In such case 2 nodes might serve 2 different pages, randomly?

(just thinking out loud)

Note:

• Why not using the term "TTL" instead of timeout, for a static page? caching terminology seems relevant here
• Also, all this makes me think of the "stale-while-revalidate"/SWR (or "cache-and-network" / Apollo), pattern looks very similar

[nimitwalia89]

@slorber this is happening for me. I have pm2 with multiple instances and page randomly starts showing 2 versions. Right now to fix this i have to redeploy the app.

Do you have any solution?

[slorber]

@nimitwalia89 unfortunately not, I don't even have a NextJS with iSSG in production, it's just something that I thought could happen. If you have the answer to this problem, I'm curious.

[erhankaradeniz]

This is a good start imho to make static sites more accessible. However it would be ideal to be able to send some kind of trigger to the server with a flag or something in those lines that a page needs to be rebuild.

isn't it a waste of resources if you'd work with only the timeout option? (not sure how much it uses)

[Nases]

Is this available with v9.3? Or is it at the proposal stage?