Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] More granular default Webpack chunking #7631

Closed
atcastle opened this issue Jun 21, 2019 · 13 comments
Closed

[RFC] More granular default Webpack chunking #7631

atcastle opened this issue Jun 21, 2019 · 13 comments
Milestone

Comments

@atcastle
Copy link
Collaborator

@atcastle atcastle commented Jun 21, 2019

Feature request

Is your feature request related to a problem? Please describe.

The current Webpack chunking strategy in Next.js is based around a ratio-based heuristic for including modules in a single "commons" chunk. Because there is very little granularity, a lot of code is either downloaded unnecessarily (because the commons chunk includes a lot of code that's not actually required for a particular route) or duplicated across multiple page bundles (because it was included in less than half of all pages).

Because these two problems are in opposition, simply adjusting the ratio will always make one better and one worse.

Describe the solution you'd like

I suggest adopting a SplitChunksPlugin configuration such as the following:

cacheGroups: {
    default: false,
    vendors: false,
    framework: {
        name: 'framework',
        chunks: 'all',
        test: /[\\/]node_modules[\\/](react|react-dom)[\\/]/,
        priority: 40
    },
    lib: {
        test(module) {
            return module.size() > 160000
        },
        name(module) {
            return /node_modules\/(.*)/.exec(module.identifier())[1]
                .replace(/\/|\\/g, "_")
        },
        priority: 30,
        minChunks: 1,
        reuseExistingChunk: true
    },
    commons: {
        name: 'commons',
        chunks: 'all',
        minChunks: totalPages,
        priority: 20
    },
    shared: {
	name: false,
        priority: 10,
        minChunks: 2,
        reuseExistingChunk: true
    }               
},
maxInitialRequests: 20

That configuration was designed with these considerations:

Greater granularity

The problems with the current chunking strategy arise from the low granularity of the page-chunk/commons-chunk paradigm. We can easily reduce code duplication by allowing SplitChunksPlugin to create more chunks. In the case of the library shared across 5 entry points, SplitChunksPlugin could simply create a chunk that only contains code shared across those 5 libraries.

Note that SplitChunksPlugin has configuration options to prevent runaway granularity, such as a situation where every module ends up in its own chunk, and the user has to download 100 chunks just to render a route. In particular, in the config above, I'm using the maxInitialRequests option to prevent the number of bundles from skyrocketing.

Cache efficiency

It should be a priority to ensure that shared modules get chunked in a way that causes minimum unnecessary cache invalidation. For an example of this principle, consider an application with 20 entry point chunks. Chunk A and Chunk B both depend on a large library and several small modules. The default SplitChunksPlugin behavior would create a single shared chunk for the intersection of Chunk A and Chunk B, containing the library and modules.

However, in this scenario, all of the following would cache-invalidate the entire chunk, including the large library:

  1. Adding a new module depended on by A and B exclusively (adds module to the chunk)
  2. Adding a new dependency from a different entry point to one of the modules or the library (module moves out of the chunk)
  3. Any change to any of the small modules shared between A and B.

For a sufficiently large library, it makes sense to bundle it all by itself to avoid potential cache invalidations, and the code sample above does exactly that.

A framework chunk

Some modules will be required for every entry point, because they are functionally part of the framework itself, such as React and React-dom. These chunks should be put into a framework chunk, which should be independent from any code introduced by the application developers--even if that code is depended on by 100% of entry points. This is because the frameworks libraries tend to be fairly large, and we know that they generally will not change except when the app developers update the version of Next.js itself. By isolating the framework code we ensure that it will not be cache-invalidated by irrelevant changes made to application code.

Describe alternatives you've considered

I've considered smaller values for maxInitialRequests, but for Next.js applications with a very large number of pages (100+) that can cause a size regression. At 20 that issue goes away. I believe that even at maxInitialRequests 20, the majority of NextJS sites will still have initial requests in the single digits. It's also not clear to me that even if it the initial requests went up to 20 that it would cause any performance degradation for users on modern browsers.

Additional context

In my initial testing on some small and medium sized Next.js apps, this enhanced webpack configuration reduces KB of total JS required for initial page load by as much as 20%. For particularly-poorly optimized sites, I think the improvement could be even greater.

@timneutkens timneutkens changed the title [RFC] More granular default Webpack [RFC] More granular default Webpack chunking Jun 22, 2019
@sokra
Copy link
Member

@sokra sokra commented Jul 21, 2019

Have you tried using splitChunks.maxSize for granular caching?

@jcruzdzk
Copy link

@jcruzdzk jcruzdzk commented Sep 7, 2019

Hi @atcastle , I am trying to split any page for a separated deploy, something like "micro-frontends", you know what I mean? Do you have any suggestion for this?

@Timer
Copy link
Member

@Timer Timer commented Jan 14, 2020

This is now on-by-default in next@canary. Releasing soon! 🚀

@Timer Timer closed this as completed Jan 14, 2020
@eltonjothi
Copy link

@eltonjothi eltonjothi commented Jan 15, 2020

wanted to test out granular chunking but i am having build issues from next@canary,
probably because i am using scss and css (@zeit/next-sass & @zeit/next-css)

Error: Encountered unknown module type: css/extract-chunks. Please open an issue.

Screenshot 2020-01-15 at 11 05 03 AM

@Timer
Copy link
Member

@Timer Timer commented Jan 15, 2020

@eltonjothi thanks for reporting! Fix is up in #10101.

Edit: should be fixed in next@^9.1.8-canary.16.

@wq93
Copy link

@wq93 wq93 commented Mar 31, 2020

Can you provide the code?

@timneutkens
Copy link
Member

@timneutkens timneutkens commented Mar 31, 2020

You can't implement this on older versions of Next.js as it requires a ton of instrumentation. You should upgrade to the latest version, no breaking changes were made between 8 and 9.

Reference #11483

@fabb
Copy link
Contributor

@fabb fabb commented Apr 1, 2020

Question: we used to keep unit tests outside of the pages folder to not make them being considered for chunking. Is this obsolete, now that granular chunking has been released?

@timneutkens
Copy link
Member

@timneutkens timneutkens commented Apr 1, 2020

No as Next.js will still turn all js/ts files in pages into entrypoints

@mbogdan0
Copy link

@mbogdan0 mbogdan0 commented May 3, 2020

Is there a way to disable that granular chunking? I wish I could reduce a number of initial http-requests of my page. Even on a very simple page with no content (plain text, no css, no plugins, anything) I have about a dozen of js chunks and http-requests.

I had to refuse using nextjs due to these reasons.

@timneutkens
Copy link
Member

@timneutkens timneutkens commented May 5, 2020

@mbogdan0 you can't disable it, you can read about the reasons behind the way it works here: https://web.dev/granular-chunking-nextjs/

It gives significantly better results than what you're proposing.

@kartikag01
Copy link

@kartikag01 kartikag01 commented Sep 2, 2020

test: /(?<!node_modules.*)[\\/]node_modules[\\/](react|react-dom|scheduler|prop-types|use-subscription)[\\/]/,

why next/dist is not a part of the framework chunk?

@balazsorban44
Copy link
Member

@balazsorban44 balazsorban44 commented Jan 29, 2022

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.

@vercel vercel locked as resolved and limited conversation to collaborators Jan 29, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests