Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include examples/js into the new Three.JS module system and rely on tree shaking #9403

Closed
bhouston opened this issue Jul 25, 2016 · 20 comments

Comments

@bhouston
Copy link
Contributor

commented Jul 25, 2016

Description of the problem

It would be great if we could move a lot of the Three.JS example/js source code into the main /src directory tree and then rely upon the module system/tree shaking to remove the code that is not needed.

This would really make it easier to add code to three.js and also make everyone's builds small at the same time. :) Basically better in every way.

I guess when one uses ThreeJS in another project one needs to rely upon a global rollup rather than a ThreeJS specific rollup. I guess that can be hard to support.

/ping @Rich-Harris

Three.js version
  • Dev
  • r79
  • ...
Browser
  • All of them
  • Chrome
  • Firefox
  • Internet Explorer
OS
  • All of them
  • Windows
  • Linux
  • Android
  • IOS
Hardware Requirements (graphics card, VR Device, ...)
@crabmusket

This comment has been minimized.

Copy link

commented May 14, 2017

How do you imagine the tree-shaking working in this case? I've been trying to get it working with just the core of Three.js but can't manage to get any benefit from it. See this question, especially the comments after the first answer. I also tried importing Three.js from src directly. After I got a transformer for the glsl files, I still observed no change in the bundle size either.

If the examples are each completely separate source trees (i.e. they aren't imported by the main library) then I guess this would be feasible - but at that point it's not really tree shaking, is it?

@mutsys

This comment has been minimized.

Copy link

commented May 26, 2017

@crabmusket I've been looking at this as well, wondering why the simplest, most minimal import of Three.js results in monstrously huge bundles containing essentially every single module. I read through the discussion you pointed to above, performed numerous experiments and went so far as to generate graphs of the Three.js dependency tree to try and get to the bottom of this. Here are a few takeaways from the experience:

Importing from src is the right way to go. This is your best bet if you are counting on static analysis tools to detect and excise any unused code.

UglifyJS couldn't do much here since it does not yet understand es6 modules.

Babili does understand es6 modules and is capable of finding unused code that can be purged.

A multi-stage build/bundling process via Webpack, transpiling from TypeScript to es6, performing code elimination via Babili, transpiling to es5 via Babel and finally minifying via UglifyJS turned out to be the most effective strategy and was the only means of making a dent in the final size of the bundle produced by Webpack.

However, try as I might, after days of trials and experimentation, I can only get the size of the Three.js derived code down to 529.96 KB and when I dig in and take a look at which modules have been included, I still see tons of code that I am not using and should not end up in the final bundle.

I haven't done enough detailed analysis to be absolutely sure just why all of the extra modules are getting pulled in, but I have some intuition here. I think this may be caused by the "convenience" exports that can be found in src/Three.js. Every module is brought up to the "surface" of Three.js in an effort to make it easier for developers to import the modules they want to use without a bunch of hunting around in the docs. Unfortunately, this seems to have the undesirable side-effect of forcibly importing all of Three.js my default into your project whether you want it or not. This wouldn't be too terrible if the uneccessary code could be removed by the usual "tree-shaking" process down the line, but it never does get removed no matter how hard you try. I suspect this is becuase this is becuase all of these pre-emptively imported all depend on one another and there is enough complexity in these dependency relationships to prevent the tree-shaking process from correctly identifying and excising unused code.

I can't state this with certainly yet, its just a hunch based on what I've observed while playing around with this for a few days. I am hoping to test this theory out soon.

@looeee

This comment has been minimized.

Copy link
Collaborator

commented May 26, 2017

Babili does understand es6 modules and is capable of finding unused code that can be purged.

Maybe I'm missing something but I don't see anywhere on the Babili page that says it can understand ES6 modules, and their online test doesn't support multiple files (compare it to the rollup repl).

Have you tested on a simple setup to see if it tree shakes unused modules?

@mutsys

This comment has been minimized.

Copy link

commented May 26, 2017

Take a look here to see some more details about what I am pursuing: babili-webpack-pugin. The authors discuss a multi-stage transpile - bundle - minify process that I've been trying out. I do have a project where I have implemented this and I can say that I do get a smaller bundle using this technique with Babili as they describe than without it.

@crabmusket

This comment has been minimized.

Copy link

commented May 26, 2017

@mutsys thanks for the in-depth analysis! At this point I am honestly just planning on forking Three and hand-removing bits I don't want. But if it's actually to do with the internal module structure of Three, it seems like it might be worth experimenting with says of improving that. The files in dist/ should obviously remain kitchen-sinkified, but if we can improve the way importing from src/ splits modules it would be a big win.

If I get time I will look into it, but as this is for work it's hard to justify making more effort than to fork, prune and move on.

@mrdoob

This comment has been minimized.

Copy link
Owner

commented May 26, 2017

I think this may be caused by the "convenience" exports that can be found in src/Three.js.

Why are you importing that file if you are building your app using modules directly? That file is only for producing the THREE.* which you shouldn't use when building with modules.

@mutsys

This comment has been minimized.

Copy link

commented May 28, 2017

@crabmusket I started working on a private fork of Three.js myself the other night to see if I could make any meaningful reduction in the final bundle size by just getting rid of the convenience exports and directly importing any necessary modules directly (as Three.js itself does internally). Since I'm also using TypeScript, this requires me to also fork and tweak up @types/three as well, making this an excellent exercise in really grokking the structure of Three.js itself. If I am able to gain any additional insight I will be sure to post back here for you.

@mrdoob Flattered that my machinations caught your attention. Here are a few data points that illustrate why I'm spending my time on this.

Importing from src/Three.js:

The contributions from Three.js total up to:

component size
stat size 729.77 KB
parsed size 529.96 KB
gzipped size 127.86 KB

and using webpack-bundle-analyzer to visualize which how much each module contributes to the final bundle, I get a very clear picture of what got pulled in from Three.js and I can see that there are many modules that I am not importing and should not be present at the end.

screen shot 2017-05-27 at 9 00 49 pm

Importing from build/three.module.js

The contributions from Three.js total up to:

component size
stat size 665.94 KB
parsed size 477.72 KB
gzipped size 120.07 KB

which is a bit smaller, but still seems to be unjustifiably large, so I would like to unxderstand just which of the Three.js modules became part of my bundle. Unfortunately, the analysis of the bundle looks like this:

screen shot 2017-05-27 at 9 18 30 pm

three.module.js is totally opaque and does not lend itself to further analysis.

Finally, importing from build/three.js

The contributions from Three.js total up to:

component size
stat size 672.68 KB
parsed size 467.73 KB
gzipped size 119.35 KB

which results in the smallest output (parsed size is what actually ends up in my webpack bundle), but one again, seems to be way larger that what I would expect to see and once again, analysis of the bundle results in:

screen shot 2017-05-27 at 9 25 34 pm

three.js is entirely opaque and thwarts any further discovery.

So although it does not result in the smallest bundle, importing from src/Three.js is the only method that allows for further analysis and provides me with any feedback on experiments.

Pretty much every other library I use provides me with view similar to what I see when I import from src/Three.js, providing me with the insight necessary to tune my use of the provided modules and make meaningful reductions in the size of my final bundle. The only two libraries that seem to be unusual or problematic in this respect are THREE and PIXI.

That being said, since I have immensely enjoyed learning and using THREE and have benefitted greatly from its existance, I am hoping that by digging into this riddle I might discover something that could lead to further improvement of an excellent library and contribute something back in return. Of course, if you are able to provide me with any insight that might help me out in my investigation, I would be grateful.

@mutsys

This comment has been minimized.

Copy link

commented May 28, 2017

@crabmusket @mrdoob OK, I spent a few hours tonight doing a deep dive into this. I was able to get a significant reduction in the size of my bundle, shaving a whole heck of a lot off of the contribution from Three.js. Here is where I am at now...

Importing from src, ignoring src/Three.js by declaring imports directly to the necessary modules (eg - import { Object3D } from "three/src/core/Object3D";) and refactoring the TypeScript definition files accordingly:

component size
stat size 435.11 KB
parsed size 323.07 KB
gzipped size 78.47 KB

screen shot 2017-05-28 at 4 44 20 am

It appears my intuition was correct and the "convenience" exports from src/Three.js do in fact interfere with tree-shaking and dead code elimination. I did not have to make any modifications to any of the Three.js source, all I did was change the way in which I imported modules in my own code.

I'm not going to claim total victory just yet until I go over this again in detail and see if I catch any mistakes in here. It is very late at night now where I am and I'm super tired so I'm not going to get too excited just yet until I get some rest and give it another once over.

That being said, I do believe that there are additional opportuinities to improve upon this further. As I was refactoring the TypeScript definition files, it was apparent that there a fair number of circular dependencies throughout Three.js, a good number of dependencies that are declared backwards/in the wrong direction and some careless coding here and there that all serves to prevent identification and removal of otherwise unnecessary code during static analysis and minification.

Some examples:

Example 1)

cameras/Camera -> math/Vector3
and
math/Vector3 -> cameras/Camera

Two strikes here:

  • circular dependency
  • having math/Vector3 depend on cameras/Camera is totally backwards

Example 2)

renderers/WebGLRenderer -> cameras/PerspectiveCamera
and
renderers/WebGLRenderer -> cameras/OrthographicCamera

not necessarily bad by itself, but take a look at what happens in the render method:

if ( background && background.isCubeTexture ) {

    if ( backgroundBoxCamera === undefined ) {

        backgroundBoxCamera = new PerspectiveCamera();

        backgroundBoxMesh = new Mesh(
            new BoxBufferGeometry( 5, 5, 5 ),
            new ShaderMaterial( {
                uniforms: ShaderLib.cube.uniforms,
                vertexShader: ShaderLib.cube.vertexShader,
                fragmentShader: ShaderLib.cube.fragmentShader,
                side: BackSide,
                depthTest: false,
                depthWrite: false,
                fog: false
            } )
        );

    }

    backgroundBoxCamera.projectionMatrix.copy( camera.projectionMatrix );

    backgroundBoxCamera.matrixWorld.extractRotation( camera.matrixWorld );
    backgroundBoxCamera.matrixWorldInverse.getInverse( backgroundBoxCamera.matrixWorld );


    backgroundBoxMesh.material.uniforms[ "tCube" ].value = background;
    backgroundBoxMesh.modelViewMatrix.multiplyMatrices( backgroundBoxCamera.matrixWorldInverse, backgroundBoxMesh.matrixWorld );

    objects.update( backgroundBoxMesh );

    _this.renderBufferDirect( backgroundBoxCamera, null, backgroundBoxMesh.geometry, backgroundBoxMesh.material, backgroundBoxMesh, null );

} else if ( background && background.isTexture ) {

    if ( backgroundPlaneCamera === undefined ) {

        backgroundPlaneCamera = new OrthographicCamera( - 1, 1, 1, - 1, 0, 1 );

        backgroundPlaneMesh = new Mesh(
            new PlaneBufferGeometry( 2, 2 ),
            new MeshBasicMaterial( { depthTest: false, depthWrite: false, fog: false } )
        );

    }

    backgroundPlaneMesh.material.map = background;

    objects.update( backgroundPlaneMesh );

    _this.renderBufferDirect( backgroundPlaneCamera, null, backgroundPlaneMesh.geometry, backgroundPlaneMesh.material, backgroundPlaneMesh, null );

}

The choice of camera here (and a few other modules as well) can only happen dynamically at runtime, so even if I never use one of these modules in my own code, both will always be included in the final bundle regardless. Sure enough, I am only only using PerspectiveCamera in my own code but I can see that my bundle contains OrthographicCamera anyway. There are quite a number of places in Three.js where similar "forced import" occurs. This example only adds about 2KB to the final bundle, but it all adds up. The area where something like this seems to be doing the most harm is in renderers/shaders. I don't know enough to really say whether or not everything in there is always needed all of the time, but I'm guessing that it isn't and some refactoring effort here might have some significant impact. There is some 100 KB or so of code there that look ripe for analysis.

@crabmusket

This comment has been minimized.

Copy link

commented May 28, 2017

@mrdoob in retrospect what you say makes things completely obvious. Maybe for other confused souls like myself it would be good to have a section in the readme or something? Just prompting people to import modules directly if they only need a subset of features. And reminding them they'll probably need a loader for .glsl files (e.g. webpack-glsl-loader).

@mutsys awesome work! Great to see those results and hear that there might be opportunities to improve further. I'll keep following this thread and will report back on the results of importing directly from src modules instead of Three.js.

My method will be basically to maintain my own local three.js file. I experimented with this before, but I was still importing everything from Three.js and relying on tree-shaking. Turns out I was putting too much faith in the system haha.

EDIT: by selectively importing files, I reduced the bundled size of Three.js from 420K to 260K. Not as much as I was hoping for, but not bad either. @mutsys I can't see where Vector3 is importing Camera?

@tschoartschi

This comment has been minimized.

Copy link

commented Dec 5, 2017

Very interesting discussion! I'll keep an eye on it since I also tried to tree shake Three.js and I was also not satisfied with the results. Maybe there will be a good solution someday soon.

@tschoartschi

This comment has been minimized.

Copy link

commented Dec 5, 2017

@mutsys I also tried something very similar to your approach. I used rollup since we already have it in our build pipeline. Do you think I can get better results with webpack. I created a repo so if anyone is interested she or he can check it out, https://github.com/tschoartschi/tree-shake-threejs

In this example there is only a rotating cube and I think the file size is still quite big. Maybe there could be better results? Would be great to get some feedback

@mutsys

This comment has been minimized.

Copy link

commented Dec 6, 2017

@tschoartschi @crabmusket Wow, it has been a while since I was tinkering around with this and haven't looked at it lately but I do still have the repo where I was working on this. As you may have gathered from my previous comments, it took a significant amount of work in my own fork to get those results and if there is some way that the effort can further our shared goal I would happy to share it with everyone for analysis and feedback.

@crabmusket Keep in mind that the analysis was performed many months ago and more recent releases may have resolved that circular dependency I indentified between math/Vector3 and cameras/Camera.

@zbigg

This comment has been minimized.

Copy link

commented Jul 10, 2018

Small followup of @mutsys work. Maybe someone will find this topic and find this usable.

Use case: bundling three.js with other app.
Environment: webpack + typescript.

This always imports whole three.js (minimal bundle size around 430k, simetimes 534k)

import { Vector3 } from 'three';
import { Vector3 } from 'three/src/math/Vector3.js';

works in JS files and, yes tree-shaking workd.
However, TypeScript build fails because '@types/three' catch only 'three' module, not files hidden in 'three/src':

TS7016: Could not find a declaration file for module 'three/src/math/Vector3.js'. (...)

My workaround:

// foo.ts
/// <reference path="three.src.d.ts"/>
import { Vector3 } from 'three/src/math/Vector3.js';
..
// three.src.d.ts
declare module 'three/src/math/Vector3.js' {
    export { Vector3 } from 'three';
}

So, i basically declare all modules from 'src' with re-exports from '@types/three'.
webpack is happy because it doesn't have to deal with 'export * ...' in src/Three.js and ts-loader is happy because it know what hides behind three/src/*.

Maybe not best solution but i can handcraft my custom three.src.d.ts for each project.

Bundle summary with workaround (TS version):

    [0] ./foo.ts + 4 modules 46.3 KiB {0} [built]
        | ./foo.ts 197 bytes [built]
        | ./node_modules/three/src/math/Vector3.js 12 KiB [built]
        | ./node_modules/three/src/math/Math.js 3.01 KiB [built]
        | ./node_modules/three/src/math/Matrix4.js 20.2 KiB [built]
        | ./node_modules/three/src/math/Quaternion.js 11 KiB [built]

Experiment source code: https://github.com/zbigg/threejs-webpack-typescript-tree-shaking

@tschoartschi

This comment has been minimized.

Copy link

commented Jul 11, 2018

@zbigg I think the problem you describe is not directly related to three.js. I think this issue comes from the TypeScript definitions. We had to do the following to prevent TypeScript from including everything:

import * as __THREE from 'three';

declare global {
    const THREE: typeof __THREE; // tslint:disable-line
}

But our real-world results on tree-shaking three.js weren't really satisfying. I think this is due to the fact how three.js is architectured.

@arodic arodic referenced this issue Jul 11, 2018
2 of 2 tasks complete
@bhouston

This comment has been minimized.

Copy link
Contributor Author

commented Jan 2, 2019

I'm going to explore solving this issue via https://github.com/systemjs/systemjs

@donmccurdy

This comment has been minimized.

Copy link
Collaborator

commented Jan 2, 2019

Related: #14803

I'm strongly in favor of converting the packages under example/js/* into ES modules and (if needed for backward-compatibility) automatically building UMD modules from those. That's high priority in my opinion.

But how would SystemJS contribute? It seems useful for end-users' applications, or keeping the example demos functional on older browsers, but I'm not sure how it relates to distribution of the threejs library as a download or npm module.

The idea of moving a majority of examples/js/* into the src/* tree is less of a slam dunk I think. If it's possible to defer that conversation until the ES module change is settled, let's go that route.

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jan 3, 2019

Some progress... #15518

@PerspectivesLab

This comment has been minimized.

Copy link

commented Jul 9, 2019

this new /jsm/ files works great for importing the examples modules,
meanwhile has someone been successful to tree shake threejs with this, out of the box ( import {...} from 'three' ), and not importing source files...
im using webpack4 enabled all options for tree shaking, but the size of threejs doesnt move, still around 700 k meanning all library is included... :(

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jul 10, 2019

I think this can now be closed.

@mrdoob mrdoob closed this Jul 10, 2019

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jul 10, 2019

@PerspectivesLab Better create a thread in the forum to discuss options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.