On-Demand ISR doesn't revalidate more than once

[zeddotes]

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

    Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:41 PDT 2021; root:xnu-7195.101.2~1/RELEASE_ARM64_T8101
    Binaries:
      Node: 14.17.1
      npm: 8.2.0
      Yarn: 1.22.10
      pnpm: N/A
    Relevant packages:
      next: 12.2.4
      eslint-config-next: N/A
      react: 17.0.2
      react-dom: 17.0.2

What browser are you using? (if relevant)

n/a

How are you deploying your application? (if relevant)

n/a

Describe the Bug

On-demand ISR is implemented (on /api/revalidate) and while it does flush the site cache once, hitting the endpoint a second time (or more) does not. This is happening in dev as well as in production. CMS is used to fetch the content and to verify this bug, I have updated the content in between calls to /api/revalidate

Expected Behavior

Cache should be flushed each time with new content from the CMS (via getStaticProps)

Link to reproduction

https://github.com/DTS-STN/Service-Canada-Labs

To Reproduce

• All pages in the app rely on getStaticProps to send a request to our CMS to get content.
• There's a revalidate API handler (/api/revalidate.js) created in our app with the following code:

const REVALIDATION_ROUTES = [
  "/home",
];

export default async function handler(req, res) {
  // Check for secret to confirm this is a valid request
  if (req.query.secret !== process.env.REVALIDATION_TOKEN) {
    return res.status(401).json({ message: "Invalid token" });
  }

  try {
    await Promise.all(REVALIDATION_ROUTES.map((route) => res.revalidate(route)));
    return res.json({ revalidated: true });
  } catch (err) {
    console.log(err);
    return res.status(500).send("Error revalidating");
  }
}

• There is no revalidate property option declared as part of the getStaticProps for any of the pages

[balasphilip]

I have a similar case, very much similar to the request above :

• I host Next.js on-premise;
• CMS pushes data into DB;
• Next.js 12.2 statically renders pages (SSG) based on data from the DB;
• then I use DB change stream events to trigger the Next.js service as soon as CMS pushes new portion of data into DB;
• as the Next.js service got triggered it pings itself by api/revalidate endpoint;
• the trick is that we have quite a few pages to re-generate, like 100+, from time to time 200+, so we use similar Promise.all approach.

The result is:

• no errors from Next.js revalidation API;
• I see some internal Next.js warning messages like "Warning: data for page "..." is 896 kB which exceeds the threshold of 128 kB" which means the revalidation is processing my page;
• new updates don't reflect in a browser

Then I use Postman and trigger api/revalidate explicitly, and the page gets updated in a browser.

Any thoughts on that?

For context: I want to use this feature to update hundreds and maybe thousands of pages in the future.Is it mature enough to rely on it or better to switch to SSR or revalidation with an interval?

Thanks!

[KurnosovNikita]

Faced the same issue, each page revalidates only once and no more. Version 12.2.0.

[ijjk]

Hi, this appears to be due to caching being done when querying from your CMS.

When revalidating content you want to be sure to skip any caches so that fresh content can be pulled. I'm gonna close this as this doesn't appear to be a bug in Next.js.

[zeddotes]

@ijjk it's actually not related to Apollo's cache.

If you check the branch bug/revalidation (https://github.com/DTS-STN/Service-Canada-Labs/blob/bug/revalidation/pages/api/revalidate.js to be exact), you'll see in /pages/api/revalidate.js that we're explicitly flushing the cache before triggering Next.js' revalidate method.

Please reopen this issue.

[ijjk]

Are you able to reproduce this without fetching from your CMS? We have tests covering this and they are passing as expected so there is most likely unexpected caching occurring preventing fresh content from being pulled from your CMS

[balasphilip]

hey team, I'm fetching from the DB right away, I don't use any CMS specific proxies/layers/APIs. Thus I'm still observing stale pages in my browser until I explicitly trigger the revalidation endpoint asking to re-generate that particular page. I'm sure everything works well on my side as the whole e2e logic works well on a small number of pages, like 20-30.

[ijjk]

@balasphilip are you on the latest version of Next.js v12.2.5? A minimal reproduction that doesn't rely on upstream data would be great to investigate this further.

[zeddotes]

@ijjk I've verified that although the issue does not happen when using fetch plainly, the moment I add that fetch to a singleton service for the purposes of caching, with an in-memory property of the service (problem area), Next.js doesn't revalidate beyond the first time. For context: I tested all permutations with and without caching (with flushing capability integrated during Next.js revalidation).

I switched to file-based memory caching of the fetch calls and was able to see the expected behaviour -- Next.js revalidated every time.

This leads me to think that Next.js is either 1) not able to properly scope services that have been imported and loses the services' references, maybe due to duplication; or 2) caches API handler responses in a way that it shouldn't...ie, passing through revalidation headers (like HIT) after a revalidation has been done once for the route.

Our original implementation was with Apollo client (for the purposes of using GraphQL) and it likely persists cache in memory, which makes sense why it was also causing issues with Next.js.

I'm all outta clues otherwise :)

[github-actions]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.