Proposal to use SSR for incremental static regeneration

[joostmeijles]

Proposal to use SSR for incremental static regeneration (https://nextjs.org/docs/basic-features/data-fetching#incremental-static-regeneration).

This is definitely no production code, but I would like to determine how incremental static regeneration could be added.

[raik]

Thanks for the work. As I am also interested in using Next.js ISR on Netlify, I tried your branch on my project. Understandable that it's a WIP, but I'll share the truncated build log here of the issue I have. It seems to fail if index.js has revalidate: 1 in it. Maybe it will help you or someone else when writing out this functionality.

If there is need do test out on a real site I'll have a small production ready project with different rendering options (as you see in the log). Let me know, would be happy to help.

9:16:14 PM: Build ready to start

[...]

9:16:45 PM: ┌─────────────────────────────┐
9:16:45 PM: │        Netlify Build        │
9:16:45 PM: └─────────────────────────────┘
9:16:45 PM: ​
9:16:45 PM: ❯ Version
9:16:45 PM:   @netlify/build 3.3.5
9:16:45 PM: ​
9:16:45 PM: ❯ Flags
9:16:45 PM:   deployId: 5f5bbeedfd9aef0007fcc129
9:16:45 PM:   mode: buildbot
9:16:45 PM: ​
9:16:45 PM: ❯ Current directory
9:16:45 PM:   /opt/build/repo
9:16:45 PM: ​
9:16:45 PM: ❯ Config file
9:16:45 PM:   /opt/build/repo/netlify.toml
9:16:45 PM: ​
9:16:45 PM: ❯ Context
9:16:45 PM:   production
9:16:48 PM: ​
9:16:48 PM: ┌────────────────────────────────────┐
9:16:48 PM: │ 1. build.command from netlify.toml │
9:16:48 PM: └────────────────────────────────────┘
9:16:48 PM: ​

[...]

9:17:19 PM: info  - Collecting page data...
9:17:22 PM: info  - Generating static pages (0/10)
9:17:25 PM: info  - Generating static pages (1/10)
9:17:25 PM: info  - Generating static pages (2/10)
9:17:25 PM: info  - Generating static pages (3/10)
9:17:25 PM: info  - Generating static pages (4/10)
9:17:26 PM: info  - Generating static pages (5/10)
9:17:26 PM: info  - Generating static pages (6/10)
9:17:26 PM: info  - Generating static pages (7/10)
9:17:26 PM: info  - Generating static pages (8/10)
9:17:27 PM: info  - Generating static pages (9/10)
9:17:28 PM: info  - Generating static pages (10/10)
9:17:28 PM: info  - Finalizing page optimization...
9:17:28 PM: Page                                                           Size     First Load JS
9:17:28 PM: ┌ ● / (ISR: 1 Seconds)                                         1.77 kB         108 kB
9:17:28 PM: ├   /_app                                                      0 B            89.2 kB
9:17:28 PM: ├ λ /[pageSlug]                                                11 kB           115 kB
9:17:28 PM: ├ ○ /404                                                       2.77 kB          92 kB
9:17:28 PM: ├ λ /awards                                                    1.61 kB         106 kB
9:17:28 PM: ├ λ /blog                                                      1.93 kB         103 kB
9:17:28 PM: ├ λ /blog/[postSlug]                                           2.42 kB         103 kB
9:17:28 PM: ├ ● /careers (ISR: 1 Seconds)                                  614 B           105 kB
9:17:28 PM: ├ ● /careers/[careerSlug]                                      1.52 kB         108 kB
9:17:28 PM: ├   └ /careers/freelancer
9:17:28 PM: ├ ● /services/[serviceSlug]                                    6.49 kB         116 kB
9:17:28 PM: ├   ├ /services/strategic-design-consulting
9:17:28 PM: ├   ├ /services/graphic-design-art-direction
9:17:28 PM: ├   ├ /services/environmental-design-wayfinding
9:17:28 PM: ├   └ [+2 more paths]
9:17:28 PM: ├ ● /team (ISR: 1 Seconds)                                     1.54 kB         106 kB
9:17:28 PM: ├ λ /work                                                      2.81 kB         109 kB
9:17:28 PM: ├ λ /work/[projectSlug]                                        11.3 kB         118 kB
9:17:28 PM: └ λ /work/disciplines/[typeSlug]                               2.89 kB         109 kB
9:17:28 PM: + First Load JS shared by all                                  89.2 kB
9:17:28 PM:   ├ chunks/05d954cf.226c58.js                                  15.7 kB
9:17:28 PM:   ├ chunks/commons.136473.js                                   10.2 kB
9:17:28 PM:   ├ chunks/fc7434783f21156e9d774eaa05d606077b9852ec.3caeca.js  12.5 kB
9:17:28 PM:   ├ chunks/framework.f4abad.js                                 40.3 kB
9:17:28 PM:   ├ chunks/main.01d169.js                                      6.37 kB
9:17:28 PM:   ├ chunks/pages/_app.e1dbc0.js                                3.36 kB
9:17:28 PM:   ├ chunks/webpack.ccf5ab.js                                   751 B
9:17:28 PM:   └ css/1d007485ba2c08ad388c.css                               5.88 kB
9:17:28 PM: λ  (Lambda)  server-side renders at runtime (uses getInitialProps or getServerSideProps)
9:17:28 PM: ○  (Static)  automatically rendered as static HTML (uses no initial props)
9:17:28 PM: ●  (SSG)     automatically generated as static HTML + JSON (uses getStaticProps)
9:17:28 PM:    (ISR)     incremental static regeneration (uses revalidate in getStaticProps)
9:17:28 PM: Redirects
9:17:28 PM: ┌ source: /:path+/
9:17:28 PM: ├ destination: /:path+
9:17:28 PM: └ permanent: true
9:17:28 PM: > velvet-website-spa@1.0.0 postbuild /opt/build/repo
9:17:28 PM: > next-on-netlify
9:17:28 PM: 🚀 Next on Netlify 🚀
9:17:28 PM:    Functions directory: out_publish/
9:17:28 PM:    Publish directory:   out_functions/
9:17:28 PM:    Make sure these are set in your netlify.toml file.
9:17:28 PM: 🌍️ Copying public/ folder to out_publish/
9:17:28 PM: 💼 Copying static NextJS assets to out_publish/
9:17:28 PM: 💫 Setting up SSR pages and SSG pages with fallback as Netlify Functions in out_functions/
9:17:28 PM:    pages/[pageSlug].js
9:17:28 PM:    pages/awards.js
9:17:28 PM:    pages/blog/[postSlug].js
9:17:28 PM:    pages/blog.js
9:17:28 PM:    pages/work/[projectSlug].js
9:17:28 PM:    pages/work/disciplines/[typeSlug].js
9:17:28 PM:    pages/work.js
9:17:28 PM:    pages/careers.js
9:17:28 PM:    pages/.js
9:17:28 PM: internal/fs/utils.js:230
9:17:28 PM:     throw err;
9:17:28 PM:     ^
9:17:28 PM: Error: ENOENT: no such file or directory, stat '.next/serverless/pages/.js'
9:17:28 PM:     at Object.statSync (fs.js:1009:3)
9:17:28 PM:     at Object.statSync (/opt/build/repo/node_modules/graceful-fs/polyfills.js:307:34)
9:17:28 PM:     at statSync (/opt/build/repo/node_modules/next-on-netlify/node_modules/fs-extra/lib/util/stat.js:10:52)
9:17:28 PM:     at getStatsSync (/opt/build/repo/node_modules/next-on-netlify/node_modules/fs-extra/lib/util/stat.js:24:19)
9:17:28 PM:     at Object.checkPathsSync (/opt/build/repo/node_modules/next-on-netlify/node_modules/fs-extra/lib/util/stat.js:49:33)
9:17:28 PM:     at copySync (/opt/build/repo/node_modules/next-on-netlify/node_modules/fs-extra/lib/copy-sync/copy-sync.js:24:38)
9:17:28 PM:     at /opt/build/repo/node_modules/next-on-netlify/lib/setupSsrPages.js:47:5
9:17:28 PM:     at Array.forEach (<anonymous>)
9:17:28 PM:     at setupSsrPages (/opt/build/repo/node_modules/next-on-netlify/lib/setupSsrPages.js:29:12)
9:17:28 PM:     at Object.<anonymous> (/opt/build/repo/node_modules/next-on-netlify/next-on-netlify.js:27:1) {
9:17:28 PM:   errno: -2,
9:17:28 PM:   syscall: 'stat',
9:17:28 PM:   code: 'ENOENT',
9:17:28 PM:   path: '.next/serverless/pages/.js'
9:17:28 PM: }
9:17:28 PM: npm ERR! code ELIFECYCLE
9:17:28 PM: npm ERR! errno 1
9:17:28 PM: npm ERR! velvet-website-spa@1.0.0 postbuild: `next-on-netlify`
9:17:28 PM: npm ERR! Exit status 1
9:17:28 PM: npm ERR!
9:17:28 PM: npm ERR! Failed at the velvet-website-spa@1.0.0 postbuild script.
9:17:28 PM: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
9:17:28 PM: npm ERR! A complete log of this run can be found in:
9:17:28 PM: npm ERR!     /opt/buildhome/.npm/_logs/2020-09-11T18_17_28_398Z-debug.log
9:17:28 PM: ​
9:17:28 PM: ┌─────────────────────────────┐
9:17:28 PM: │   "build.command" failed    │
9:17:28 PM: └─────────────────────────────┘
9:17:28 PM: ​
9:17:28 PM:   Error message
9:17:28 PM:   Command failed with exit code 1: npm run build
9:17:28 PM: ​
9:17:28 PM:   Error location
9:17:28 PM:   In build.command from netlify.toml:
9:17:28 PM:   npm run build
9:17:28 PM: ​
9:17:28 PM:   Resolved config
9:17:28 PM:   build:
9:17:28 PM:     command: npm run build
9:17:28 PM:     commandOrigin: config
9:17:28 PM:     environment:
9:17:28 PM:       - NEXT_PUBLIC_WP_API_BASE
9:17:28 PM:     functions: /opt/build/repo/out_functions
9:17:28 PM:     publish: /opt/build/repo/out_publish

[...]

9:17:28 PM: Error running command: Build script returned non-zero exit code: 1
9:17:28 PM: Failing build: Failed to build site
9:17:28 PM: Failed during stage 'building site': Build script returned non-zero exit code: 1
9:17:28 PM: Finished processing build request in 1m11.253303619s

[FinnWoelm]

Hi @joostmeijles,

Thank you so much for this PR! I think server-side rendering all pages that use revalidate is a good temporary workaround until we have a solution that can more closely match the intended behavior.

Your code is a fantastic start. I need to write some tests (thanks for already pointing out an issue, @raik). I also want to check for some edge cases, but then it's definitely good to go. I'm hoping that we can get this tested and merged in the next 48 hours. 🚀

I'll keep you updated!

---

PS: @raik, if you want to try it out today: I believe the fix for the issue you encountered is to replace line 81 in allNextJsPages.js:

      const filePath = join("pages", `${route}.js`);

with

      const filePath = join("pages, `${route.replace(/^\/$/, "/index")}.js`)

For pages/index.js, the route is /. So we need to manually convert it back to /index, so that it's the correct file path.

[FinnWoelm]

Hi all,

Based on the excellent work done in this PR, I created a WIP branch to server-side render pages with incremental static regeneration: incremental-static-regeneration. It looks like it's working well — but I would like to refactor some code before we merge it into master. So it may be another week before it ships.

@raik and others: Do you want to try it out and let us know how it goes? npm install netlify/next-on-netlify#incremental-static-regeneration

[joostmeijles]

@FinnWoelm great work!

Edit: my (deleted) posted error is definitely unrelated.

[thomkrupa]

@FinnWoelm tested it on a production website and works like a charm!

[s-kris]

Can you add about this ISG support in the ReadMe?