Unexpected bundle Buffer ployfill into client chunks

[chentsulin]

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
Platform: darwin
Arch: arm64
Version: Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:24 PDT 2021; root:xnu-8019.41.5~1/RELEASE_ARM64_T8101
Binaries:
Node: 16.15.1
npm: 8.11.0
Yarn: 1.22.18
pnpm: N/A
Relevant packages:
next: 12.2.5
eslint-config-next: 12.2.5
react: 18.2.0
react-dom: 18.2.0

What browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

Describe the Bug

A typeof Buffer check, for example:

const isNode = typeof Buffer !== 'undefined';
console.log({ isNode });

It causes a full Buffer ployfill (22.51Kb) to be put into the client bundle.

Expected Behavior

A typeof Buffer check should just return 'undefined' in browsers instead.

Link to reproduction

https://github.com/chentsulin/nextjs-issue-40178-bundle-client-buffer

To Reproduce

Add the following lines to the pages/index.js file:

const isNode = typeof Buffer !== 'undefined';
console.log({ isNode });

[SukkaW]

It is because Next.js can not identify if you are actually using the Buffer or not.

The proper way of detecting non-browser should be typeof window === 'undefined'.

[chentsulin]

That just a simple way to demonstrate the problem.

In the real case, we use sindresorhus/serialize-error in our app and our bundle size unexpectedly gained 22.51Kb due to these lines in the library:

https://github.com/sindresorhus/serialize-error/blob/a212a8c3902fa1ff1fdef8f7625dd0cc6d6e89a1/index.js#L97-L100

[SukkaW]

@chentsulin It seems that the serialize-error should only be used on the server-side, not to use on the client-side.

[chentsulin]

I believe serialize-error is designed to be used both in browser and server. It handles DOMException as well:
https://github.com/sindresorhus/serialize-error/blob/a212a8c3902fa1ff1fdef8f7625dd0cc6d6e89a1/error-constructors.js#L10-L11

If it is designed to be used only on the server-side, it doesn't need to check Buffer exists or not at the first place.

And the weekly downloads of this package almost hits 5M. A lot of people including us already use it with redux
ref: reduxjs/redux-toolkit#2392

[chentsulin]

I found a ProvidePlugin that may cause this problem:

next.js/packages/next/build/webpack-config.ts

Lines 1664 to 1666 in 3466862

	new webpack.ProvidePlugin({
	// Buffer is used by getInlineScriptSource
	Buffer: [require.resolve('buffer'), 'Buffer'],

[Fonger]

The comment indicates that the Buffer polyfill is for getInlineScriptSource however it is only used by next.js at server side. (exported at Next/Document)

next.js/packages/next/pages/_document.tsx

Line 923 in ca807d6

static getInlineScriptSource(context: Readonly<HtmlProps>): string {

IMO it's not necessary to polyfill Buffer for server-side code because Buffer had been available since node.js 0.10.x ref

If this polyfill is intended for edge runtime, I think we should find a way to fix this issue for client side.

[goustaff]

Hello,
I think I have a similar issue.

Using Buffer on the client-side results in a ~ +8kB first-load JS.
I'm using Next v14.1.4.

What should I do?

EDIT: I will just "Workaround" it since I have a total control of the use of Buffer in my current implementation, and I'm just going to avoid it... Buuuut... This issue is quite problematic imo.

EDIT2: omg, I'm confused. Buffer is not supported by browsers, but ArrayBuffer is.
https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer

[kirstymullen]

I found a ProvidePlugin that may cause this problem:

next.js/packages/next/build/webpack-config.ts

Lines 1664 to 1666 in 3466862

	new webpack.ProvidePlugin({
	// Buffer is used by getInlineScriptSource
	Buffer: [require.resolve('buffer'), 'Buffer'],

This does seem to be the culprit. I would like to remove Buffer from our client bundle as part of an optimisation strategy.

The only usages in my code base are these functions:

const base64encode = (string: string) => {
    if (typeof window !== `undefined`) {
        return btoa(string)
    } else {
        return Buffer.from(string, `utf8`).toString(`base64`)
    }
}

const base64decode = (string: string) => {
    if (typeof window !== `undefined`) {
        return atob(string)
    } else {
        return Buffer.from(string, `base64`).toString(`utf8`)
    }
}

If I remove the use of Buffer here, then it disappears from my client bundle.

Is there something I'm missing for Next.js to be able to correctly detect usage on the client and not polyfill? I searched discussions and typeof window !== 'undefined' seemed to be the recommended check.

Any help would be appreciated.

[goustaff]

I found a ProvidePlugin that may cause this problem:

next.js/packages/next/build/webpack-config.ts

Lines 1664 to 1666 in 3466862

	new webpack.ProvidePlugin({
	// Buffer is used by getInlineScriptSource
	Buffer: [require.resolve('buffer'), 'Buffer'],

This does seem to be the culprit. I would like to remove Buffer from our client bundle as part of an optimisation strategy.

The only usages in my code base are these functions:

const base64encode = (string: string) => {
    if (typeof window !== `undefined`) {
        return btoa(string)
    } else {
        return Buffer.from(string, `utf8`).toString(`base64`)
    }
}

const base64decode = (string: string) => {
    if (typeof window !== `undefined`) {
        return atob(string)
    } else {
        return Buffer.from(string, `base64`).toString(`utf8`)
    }
}

If I remove the use of Buffer here, then it disappears from my client bundle.

Is there something I'm missing for Next.js to be able to correctly detect usage on the client and not polyfill? I searched discussions and typeof window !== 'undefined' seemed to be the recommended check.

Any help would be appreciated.

I got confused by this too.

Buffer is NOT a front-end API. Buffer is a Node.js BACK-END implementation.
Then, when you use Buffer in the front-end, Next.js polyfills it because it's required to run on the front-end.

To be the most confusing as possible, the standards are:

• Buffer is a Node.js API, designed for back-end purposes: https://nodejs.org/api/buffer.html
• ArrayBuffer is a front-end object, designed to be used in a client: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer

Then, it is ArrayBuffer which is native and that will not produce bundle size inflation.
Maybe there's also other front-end built-ins to handle buffers, idk.

[kirstymullen]

I found a ProvidePlugin that may cause this problem:

next.js/packages/next/build/webpack-config.ts

Lines 1664 to 1666 in 3466862

	new webpack.ProvidePlugin({
	// Buffer is used by getInlineScriptSource
	Buffer: [require.resolve('buffer'), 'Buffer'],

This does seem to be the culprit. I would like to remove Buffer from our client bundle as part of an optimisation strategy.
The only usages in my code base are these functions:

const base64encode = (string: string) => {
    if (typeof window !== `undefined`) {
        return btoa(string)
    } else {
        return Buffer.from(string, `utf8`).toString(`base64`)
    }
}

const base64decode = (string: string) => {
    if (typeof window !== `undefined`) {
        return atob(string)
    } else {
        return Buffer.from(string, `base64`).toString(`utf8`)
    }
}

If I remove the use of Buffer here, then it disappears from my client bundle.
Is there something I'm missing for Next.js to be able to correctly detect usage on the client and not polyfill? I searched discussions and typeof window !== 'undefined' seemed to be the recommended check.
Any help would be appreciated.

I got confused by this too.

Buffer is NOT a front-end API. Buffer is a Node.js BACK-END implementation. Then, when you use Buffer in the front-end, Next.js polyfills it because it's required to run on the front-end.

To be the most confusing as possible, the standards are:

• Buffer is a Node.js API, designed for back-end purposes: https://nodejs.org/api/buffer.html
• ArrayBuffer is a front-end object, designed to be used in a client: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer

Then, it is ArrayBuffer which is native and that will not produce bundle size inflation. Maybe there's also other front-end built-ins to handle buffers, idk.

I know that Buffer is not available on the front end, which is why I use btoa and atob when the window is available. So, Buffer will never ever be called by client-side code in my application.

The issue here is that it is not possible to opt out of Buffer being polyfilled when you 100% know that it will not be called on the client.

Given that this issue has been open for ages, I'm likely to just switch to the 'legacy' atob and btoa Node.js functions for the server. I don't want an ArrayBuffer as all I'm doing is decoding/encoding a base64 string.

[goustaff]

I found a ProvidePlugin that may cause this problem:

next.js/packages/next/build/webpack-config.ts

Lines 1664 to 1666 in 3466862

	new webpack.ProvidePlugin({
	// Buffer is used by getInlineScriptSource
	Buffer: [require.resolve('buffer'), 'Buffer'],

This does seem to be the culprit. I would like to remove Buffer from our client bundle as part of an optimisation strategy.
The only usages in my code base are these functions:

const base64encode = (string: string) => {
    if (typeof window !== `undefined`) {
        return btoa(string)
    } else {
        return Buffer.from(string, `utf8`).toString(`base64`)
    }
}

const base64decode = (string: string) => {
    if (typeof window !== `undefined`) {
        return atob(string)
    } else {
        return Buffer.from(string, `base64`).toString(`utf8`)
    }
}

If I remove the use of Buffer here, then it disappears from my client bundle.
Is there something I'm missing for Next.js to be able to correctly detect usage on the client and not polyfill? I searched discussions and typeof window !== 'undefined' seemed to be the recommended check.
Any help would be appreciated.

I got confused by this too.
Buffer is NOT a front-end API. Buffer is a Node.js BACK-END implementation. Then, when you use Buffer in the front-end, Next.js polyfills it because it's required to run on the front-end.
To be the most confusing as possible, the standards are:

• Buffer is a Node.js API, designed for back-end purposes: https://nodejs.org/api/buffer.html
• ArrayBuffer is a front-end object, designed to be used in a client: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer

Then, it is ArrayBuffer which is native and that will not produce bundle size inflation. Maybe there's also other front-end built-ins to handle buffers, idk.

I know that Buffer is not available on the front end, which is why I use btoa and atob when the window is available. So, Buffer will never ever be called by client-side code in my application.

The issue here is that it is not possible to opt out of Buffer being polyfilled when you 100% know that it will not be called on the client.

Given that this issue has been open for ages, I'm likely to just switch to the 'legacy' atob and btoa Node.js functions for the server. I don't want an ArrayBuffer as all I'm doing is decoding/encoding a base64 string.

Hmmm...
Maybe typeof window !== "undefined" is not the better way to get the Buffer polyfill opt-out from the bundle size...

Idk if you're using App or Pages Router.
On App Router, I'm used to use some use client magics which opt-out nicely the undesired code. Sometimes, I also need to split my files to help Webpack a bit.

If no solution really works, perhaps the most “Ugly” hack to do would be to lazy load the component in some inelegant way, completely fooling Next.js.
https://v3.gatsbyjs.com/docs/mdx/importing-and-using-components/#lazy-loading-components

[kirstymullen]

I found a ProvidePlugin that may cause this problem:

next.js/packages/next/build/webpack-config.ts

Lines 1664 to 1666 in 3466862

	new webpack.ProvidePlugin({
	// Buffer is used by getInlineScriptSource
	Buffer: [require.resolve('buffer'), 'Buffer'],

This does seem to be the culprit. I would like to remove Buffer from our client bundle as part of an optimisation strategy.
The only usages in my code base are these functions:

const base64encode = (string: string) => {
    if (typeof window !== `undefined`) {
        return btoa(string)
    } else {
        return Buffer.from(string, `utf8`).toString(`base64`)
    }
}

const base64decode = (string: string) => {
    if (typeof window !== `undefined`) {
        return atob(string)
    } else {
        return Buffer.from(string, `base64`).toString(`utf8`)
    }
}

If I remove the use of Buffer here, then it disappears from my client bundle.
Is there something I'm missing for Next.js to be able to correctly detect usage on the client and not polyfill? I searched discussions and typeof window !== 'undefined' seemed to be the recommended check.
Any help would be appreciated.

I got confused by this too.
Buffer is NOT a front-end API. Buffer is a Node.js BACK-END implementation. Then, when you use Buffer in the front-end, Next.js polyfills it because it's required to run on the front-end.
To be the most confusing as possible, the standards are:

• Buffer is a Node.js API, designed for back-end purposes: https://nodejs.org/api/buffer.html
• ArrayBuffer is a front-end object, designed to be used in a client: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer

Then, it is ArrayBuffer which is native and that will not produce bundle size inflation. Maybe there's also other front-end built-ins to handle buffers, idk.

I know that Buffer is not available on the front end, which is why I use btoa and atob when the window is available. So, Buffer will never ever be called by client-side code in my application.
The issue here is that it is not possible to opt out of Buffer being polyfilled when you 100% know that it will not be called on the client.
Given that this issue has been open for ages, I'm likely to just switch to the 'legacy' atob and btoa Node.js functions for the server. I don't want an ArrayBuffer as all I'm doing is decoding/encoding a base64 string.

Hmmm... Maybe typeof window !== "undefined" is not the better way to get the Buffer polyfill opt-out from the bundle size...

Idk if you're using App or Pages Router. On App Router, I'm used to use some use client magics which opt-out nicely the undesired code. Sometimes, I also need to split my files to help Webpack a bit.

If no solution really works, perhaps the most “Ugly” hack to do would be to lazy load the component in some inelegant way, completely fooling Next.js. https://v3.gatsbyjs.com/docs/mdx/importing-and-using-components/#lazy-loading-components

It's the same regardless of whether it's the pages router or app router (I'm in the process of migrating a large application to the app router). I'm just going to switch to the atob and btoa methods that have both server and client implementations.

Workarounds are great, but it'd be better if Next.js allowed proper control over this specific package being polyfilled or not. It's only an issue with Buffer.

[philwolstenholme]

I've just ran into this too while trying to optimise my bundle size and switch to https://es-toolkit.slash.page.

es-toolkit supports working with native Node buffers, and that seems to have triggered a polyfill to be added to our client-side bundle size reports (the reports are only generated when the isServer Webpack context is false) and our total bundle size has actually increased!

I suppose I'll have to work out what client side code is using an es-toolkit function that supports buffers and use something else to avoid this happening.

[philwolstenholme]

No idea how sensible this is yet (especially as it came straight out of Claude and I've not yet put it through its paces), but for anyone else with this issue I added this to my webpack config to be able to remove the buffer polyfill added by Next. This assumes you have no reason to need the polyfill in client side bundles!

After adding this I could use es-toolkit imports without triggering the polyfill being added.

 webpack: (config, { webpack, buildId, dev, isServer }) => {
   if (!isServer) {
      // Remove the buffer fallback from resolve.fallback
      if (config.resolve.fallback) {
        delete config.resolve.fallback.buffer;
      }

      // Remove Buffer from ProvidePlugin
      const plugins = config.plugins || [];

      for (let i = plugins.length - 1; i >= 0; i--) {
        const plugin = plugins[i];
        if (plugin.constructor.name === 'ProvidePlugin') {
          const definitions = plugin.definitions;
          if (definitions && definitions.Buffer) {
            delete definitions.Buffer;

            // If the plugin has no more definitions, remove it entirely
            if (Object.keys(definitions).length === 0) {
              plugins.splice(i, 1);
            }
          }
        }
      }
    }

    return config;
  }