Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tree-shaking with lodash-es #1750

Closed
jdalton opened this Issue Dec 12, 2015 · 43 comments

Comments

Projects
None yet
@jdalton
Copy link
Contributor

jdalton commented Dec 12, 2015

See #1612 (comment).
Currently tree-shaking lodash-es modules is not possible with webpack 2 beta.

First clone into node_modules:

git clone -b es --depth=1 https://github.com/lodash/lodash.git ./node_modules/lodash-es

then:

npm i -g webpack@beta

then:

npm i babel-loader@^5 babel-core@^5

with test.js:

import { chunk } from 'lodash-es';
console.log(chunk([1,2,3,4], 2));

then:

webpack ./test.js ./bundle.js -p --display-chunks --display-modules --display-origins --module-bind js=babel-loader?blacklist[]=es6.modules

is

> gzip-size bundle.js | pretty-bytes
30.41 kB

While the output of import { chunk } from 'lodash-es/array/chunk'
is

> gzip-size bundle.js | pretty-bytes
1.19 kB

Related to rollup/rollup#45

@shama

This comment has been minimized.

Copy link
Member

shama commented Dec 27, 2015

@jdalton Thanks for the notice! I'm sure @sokra will address when he gets time for it.

@jdalton

This comment has been minimized.

Copy link
Contributor Author

jdalton commented Dec 27, 2015

Anything I can do to help?

@shama

This comment has been minimized.

Copy link
Member

shama commented Dec 27, 2015

Yes! If you want, please test out the webpack 2 beta and investigate any issues you find. Or take a look at the issue tracker to help free up more of @sokra's time to put towards webpack 2. There are many questions that just need a little bit of research to address. Thanks!

@jdalton

This comment has been minimized.

Copy link
Contributor Author

jdalton commented Dec 27, 2015

Yes! If you want, please test out the webpack 2 beta and investigate any issues you find.

This issue is about webpack 2 beta, I did the investigation, and outlined how to reproduce the issue in detail. I believe I've created modules in a way that should enable tree-shaking. @shama do you see any obvious problems with my approach? The only thing I can think of is that the inclusion of

export { default as default } from './lodash.default';

is giving webpack 2 grief. What do you think?

@shama

This comment has been minimized.

Copy link
Member

shama commented Dec 27, 2015

I'm not sure what the cause of this issue is. If I were to take a guess, it's likely babel is transforming the modules in a way that webpack can't remove the unused code or webpack is tripping on some syntax in lodash-es.

I'm not sure, I haven't tried debugging this issue yet.

@jdalton

This comment has been minimized.

Copy link
Contributor Author

jdalton commented Dec 27, 2015

If I were to take a guess, it's likely babel is transforming the modules in a way that webpack can't remove the unused code or webpack is tripping on some syntax in lodash-es.

\cc @sebmck

@sparty02

This comment has been minimized.

Copy link

sparty02 commented Dec 27, 2015

@jdalton can you try with Babel 6? default export semantics changed between Babel 5 and Babel 6, wondering if that could impact it?

@kittens

This comment has been minimized.

Copy link

kittens commented Dec 27, 2015

Is Webpack doing the DCE on the CommonJS output?

@sokra

This comment has been minimized.

Copy link
Member

sokra commented Dec 27, 2015

Not babels fault. I noticed the issue, but it's a bit difficult.

We can optimize it, but this would be an unsafe optimization. The ES6 spec says that all dependencies must be loaded even if the exports are not used.

@jdalton

This comment has been minimized.

Copy link
Contributor Author

jdalton commented Dec 27, 2015

@sparty02

can you try with Babel 6?

Just tried Babel 6 and the result is similar to Babel 5 in terms of file size.

@sokra

but it's a bit difficult.

Can you clarify (deep dive) into what the tricky part is.

The ES6 spec says that all dependencies must be loaded even if the exports are not used.

Isn't the point of the webpack tree shake optimization to work around this?
Is there something I can do with how I generate lodash-es modules to make tree-shaking easier for you?

@indeyets

This comment has been minimized.

Copy link

indeyets commented Feb 19, 2016

@jdalton generating single file without re-exports would solve the issue, I believe

@jdalton

This comment has been minimized.

Copy link
Contributor Author

jdalton commented Feb 19, 2016

@indeyets Would you give an example/gist to clarify?

@Awk34

This comment has been minimized.

Copy link

Awk34 commented Feb 24, 2016

I think what @indeyets is suggesting is instead of importing and re-exporting like is done now:

...
export { default as add } from './add';
...

to concatenate everything into a single file like this:

...
export function add(augend, addend) {
  var result;
  if (augend === undefined && addend === undefined) {
    return 0;
  }
  if (augend !== undefined) {
    result = augend;
  }
  if (addend !== undefined) {
    result = result === undefined ? addend : (result + addend);
  }
  return result;
}
...

I doubt that this would change anything, though.

@jdalton

This comment has been minimized.

Copy link
Contributor Author

jdalton commented Feb 24, 2016

@Awk34

I doubt that this would change anything, though.

Yeah. If it's already having a hard time recognizing there are no size effects to something straight forward (as it is now) I can't imagine inlining packages (with their local vars) would make things easier.

@Akkuma

This comment has been minimized.

Copy link
Contributor

Akkuma commented Mar 25, 2016

Maybe I'm misunderstanding the issue, but wouldn't that have to work @Awk34 ? Otherwise the entire tree shaking system would clearly not be working. The issue from my understanding is that by re-exporting causes webpack to not safely know if the original file causes a side effect during import. In other words, much like you can cause side effects just by require('something'), the initial imports to then export could be causing side effects. If every method was inlined into one file there would be no ambiguity.

This still seems highly impractical for most developers as they don't have tooling to make an inlined file just for webpack. My suggestion for @sokra would be to allow explicitly marking files as having side effects that would work around the spec. Webpack already has its own require.ensure, so would it be that much of a leap to be able to declare imports with side effects in the off chance you actually do?

@nlwillia

This comment has been minimized.

Copy link

nlwillia commented Mar 25, 2016

It would be nice if there were standards support for a "weak reexport" that would say "I'm exporting this thing, but it doesn't need to be evaluated unless someone uses it". This would not change the semantics of loading except that a loader could use it as a signal to do more aggressive optimization. Nothing prevents loaders from evolving this on their own as a comment or annotation or external configuration, but if there's value to it then having it in the language instead of being loader-specific would be better for everyone.

Obviously, if a naive weak reexport targets a module with default side-effects then you might see inconsistent behavior, but I don't think that risk means that a well-structured library like lodash-es shouldn't have the option to declare its own safety. In SystemJS where all this happens in the browser, it's rather important that reexports not be eagerly fetched, but there's no way to avoid that today without drilling down to the individual files (import {x} from 'lodash-es/x') which is quite a bit less pleasant syntactically than plucking names out of a single lodash import (import {x,y,z} from 'lodash-es'). Webpack doesn't have to be as concerned with when/if the dependency gets fetched (not that reading hundreds of lodash modules that it doesn't need to is a great thing), but it still would benefit from a clear signal that it can exclude something from the bundle.

@cspotcode

This comment has been minimized.

Copy link

cspotcode commented Mar 26, 2016

LoDash uses its own functions to create some of its other functions, which triggers mandatory bundling of a lot of unused code. For example, instantiating without invokes rest. This means that, even if without is never imported, the entire implementation of rest and all its dependencies must be packaged in order to preserve possible side-effects. Tree shaking and dead code elimination aren't allowed to remove rest unless they can prove it's a pure function (no side-effects). Because JS is so dynamic and doesn't have static typing, this is almost never possible to prove at build time.

Dead code removers like Uglify could theoretically have a "frozen environment" or "predictable builtins" mode where they assume that all JS built-ins (Object, RegExp, window, etc) follow the JS engine's native behavior. That means it could make additional assumptions about what does and doesn't trigger side-effects.

For example, today when Uglify sees this:

const unused = Math.floor;

it can only minify to this:

Math.floor; // gotta keep those side-effects; "floor" might have a getter function

With a "frozen environment" option, however, it could assume that accessing Math.floor is side-effect-free because no one has monkey-patched the environment. This would require the code minifier to contain an inventory of JS's native APIs so it can make those decisions. Anything not in that inventory, such as a new browser API or node built-in, would be assumed to trigger side-effects.

@cspotcode

This comment has been minimized.

Copy link

cspotcode commented Mar 26, 2016

I put together a ready-to-run project that shows what webpack is emitting before and after dead code elimination. It's more grokkable than the default minified output. It has certainly helped me thinking about which code is reachable, what triggers side-effects, etc.

https://github.com/cspotcode/lodash-webpack-DCE-example

It writes plain and gzipped file sizes into build/file-sizes.txt. Webpack emits code comments marking all exports as either used or unused.

@Jessidhia

This comment has been minimized.

Copy link
Member

Jessidhia commented Oct 4, 2016

This is still the case in beta.25.

EDIT: I see now that all this was discussed before way in the middle of the thread; sorry for the noise :(

While webpack 2 will, correctly, not re-export any of the unused top-level re-exports, the __webpack_require__ calls are left in. While this will keep correctness as all the side-effects of the modules (such as requiring other modules and calling their functions) will still happen, it is not deletable by uglify (the entry point from @cspotcode's repo, the same module after uglify without mangling).

ryan-roemer added a commit to FormidableLabs/radium that referenced this issue Feb 5, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (#965)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects

ryan-roemer added a commit to FormidableLabs/component-playground that referenced this issue Feb 5, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (#121)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects

tptee added a commit to FormidableLabs/redux-little-router that referenced this issue Feb 5, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles.

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* #262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects
@edmorley

This comment has been minimized.

Copy link
Member

edmorley commented Mar 12, 2018

I think this issue is now resolved, so can be closed?

ryan-roemer added a commit to FormidableLabs/spectacle that referenced this issue Mar 15, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (#465)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects

ryan-roemer added a commit to FormidableLabs/formidable-charts that referenced this issue Mar 15, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (#14)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects

ryan-roemer added a commit to FormidableLabs/react-animations that referenced this issue Mar 15, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (#24)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects

ryan-roemer added a commit to FormidableLabs/react-game-kit that referenced this issue Mar 15, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (#59)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects

ryan-roemer added a commit to FormidableLabs/react-music that referenced this issue Mar 15, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (#51)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects
@malliapi

This comment has been minimized.

Copy link

malliapi commented May 13, 2018

is it ? How do we do tree-shaking with lodash-es and webpack 4?

@jdalton jdalton closed this May 13, 2018

@jdalton

This comment has been minimized.

Copy link
Contributor Author

jdalton commented May 13, 2018

Yep!

Webpack 4 supports the "sideEffects": false flag, which lodash-es uses to enables cherry-picking.

@LKay

This comment has been minimized.

Copy link

LKay commented Jun 7, 2018

@jdalton Can you provide any working example? I have usual es6 imports ie. import { get } from "lodash-es" and webpack is not tree shaking the lodash at all resulting in including full lodash library into my bundle. I'm using Webpack 4.11 so sideEffects flag should be in use, but it's not. Only explicit functions import ie. import get from "lodash-es/get" make the bundle smaller.

@AviVahl

This comment has been minimized.

@LKay

This comment has been minimized.

Copy link

LKay commented Jun 7, 2018

@AviVahl Thank you, but I import only specific functions from lodash-es (ie/ import { isEmpty, isNil } from "lodash-es") not all as combined object (import * as _ from "lodash-es"). And tree shaking doesn't work if cherry picking is in place.

@sheepsteak

This comment has been minimized.

Copy link

sheepsteak commented Jun 7, 2018

@LKay if you're using Babel make sure it isn't transpiling the ES6 import into require. Webpack needs to see the import and exports statements to do tree-shaking. If you're using babel-preset-env you can turn off the transpiling like this:

{
  "presets": [
    ["env", {
      "modules": false
    }]
  ]
}

jaredbrookswhite added a commit to jaredbrookswhite/radium that referenced this issue Aug 22, 2018

`webpack@4` / `webpack@next` will support `package.json:sideEffects: …
…false` wherein libraries can indicate their ESM re-exports are side effect free and can be much more efficiently removed for smaller, faster final bundles. (FormidableLabs#965)

Lodash has already rolled out this change in
https://unpkg.com/lodash-es@4.17.5/package.json

## Issues

This was originally uncovered / discussed at length in:

* webpack/webpack#1750

This PR should resolve the issues discussed in:

* FormidableLabs/victory#549
* FormidableLabs/redux-little-router#262

## Changes

* Add `sideEffects: false` to `package.json` to allow webpack4 tree-shaking to actually remove all unused code.

> This PR has been automatically opened by your friendly [`multibot`](https://github.com/FormidableLabs/multibot/). The transform code and documentation is available at: https://github.com/FormidableLabs/multibot-examples/tree/master/transforms/webpack-side-effects
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.