Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Javascript Bundling produces huge 13MB js files #4506

Closed
VegasJames opened this issue May 14, 2016 · 82 comments
Closed

Javascript Bundling produces huge 13MB js files #4506

VegasJames opened this issue May 14, 2016 · 82 comments

Comments

@VegasJames
Copy link

@VegasJames VegasJames commented May 14, 2016

Steps to reproduce

  1. Install Magento via composer. v2.0.5
  2. Use admin to enable Javascript Bundling. (Stores -> Configuration -> Advanced -> Developer -> JavaScript Settings) and set Merge JavaScript Files and Enable Javascript Bundling to Yes
  3. Left "Minify JavaScript Files" to No so the file could be inspected, but even with it as Yes the bundle0.min.js files is 8MB.
  4. Had to run bin/magento setup:di:compile
  5. Then did bin/magento setup:static-content:deploy
  6. Observe the size of the generated bundle0.js file.

Expected result

Before bundling Chrome showed about 515KB of javascript being loaded, bundling the files in to one should result in a file of a similar size, not 25 times as big.

Actual result
ls -lh pub/static/frontend/Magento/blank/en_US/js/bundle/
total 13M
-rw-rw-rw-. 1 magento apache 13M May 14 07:23 bundle0.js

I've investigated the content of the file somewhat. After a lot of valid javascript we get to this bit:
//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJz (truncated)
This goes on and on with random characters for 1,414,372 bytes.
The rest of the file then appears to be valid javascript, so unfortunately the above base64 only explains about 10% of the bloat, so I'm not sure where the rest comes from, but certainly this seems wrong.

Additionally, this is more of a feature request, but can the bundling leave out jquery and jquery-ui?
These are the two largest files accounting for 700KB of javascript and almost 200KB of data transferred, and they also could be fetched from Google CDN instead:
https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js
https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.4/jquery-ui.min.js

@pboisvert
Copy link

@pboisvert pboisvert commented May 14, 2016

Chuck--can you get the team to investigate given the potential for performance impact if reproducible?

@rkhoury82
Copy link

@rkhoury82 rkhoury82 commented May 16, 2016

I also suffered the same problem on the same version installed via composer.

@viktor-zhuromskyy
Copy link

@viktor-zhuromskyy viktor-zhuromskyy commented May 16, 2016

Same thing here. 👎

@viktor-zhuromskyy
Copy link

@viktor-zhuromskyy viktor-zhuromskyy commented May 16, 2016

Not even that! On 2.0.5 after all static content is compiled without any errors, there is a fatal error on whole Magento installation dues to errors in DOM bundling. With the JS Bundling OFF everything works normal.

Please see screenshots.
image 2016-05-16 17 19 12
image 2016-05-16 17 19 17

@choukalos
Copy link

@choukalos choukalos commented May 17, 2016

I'll check with the team re bundle size. I believe we implemented a very simple bundling algorithm so it's possible we're pulling in some additional files in the bundle.

Is anyone else seeing bundle.js errors on 2.0.5? @devdesco-ceo did you create a separate Github issue for the bundle error on 2.0.5?

@vkorotun
Copy link
Contributor

@vkorotun vkorotun commented May 18, 2016

@VegasJames It seems like your theme doesn't inherit from Blank or Luma Magento native themes, so exclude list for JS files bundling is missed. You need to configre the exclude list for your custom theme. As a reference example you can have a look into configuration for Blank or Luma theme:

  • /app/design/frontend/Magento/blank/etc/view.xml
  • /app/design/frontend/Magento/luma/etc/view.xml

See "exclude" node content.

@choukalos
Copy link

@choukalos choukalos commented May 18, 2016

@VegasJames Please close or let us know if that didn't address your issue.
@devdesco-ceo What themes do you have installed?

@viktor-zhuromskyy
Copy link

@viktor-zhuromskyy viktor-zhuromskyy commented May 18, 2016

I have Porto theme installed and my theme is a child theme of Porto which is based on Blank.
my /etc/view.xml file has all proper exclude statements for the JS files.

image 2016-05-18 13 02 16

@VegasJames
Copy link
Author

@VegasJames VegasJames commented May 19, 2016

incorrect, this is a stock Magento install. the bundle file inside both the
blank and Luma theme directories is 13MB

On Wednesday, May 18, 2016, Chuck Choukalos notifications@github.com
wrote:

@VegasJames https://github.com/VegasJames Please close or let us know
if that didn't address your issue.
@devdesco-ceo https://github.com/devdesco-ceo What themes do you have
installed?


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4506 (comment)

@TommyKolkman
Copy link

@TommyKolkman TommyKolkman commented May 25, 2016

I can second VegasJames's problem on 2.0.7. Also, my theme inherits correctly.

@atIOCrON
Copy link

@atIOCrON atIOCrON commented Jun 4, 2016

@vkorotun we have the same issue verified with Magento 2.0.6 using the default Luma theme:

Without JS Bundling (453.8kb): http://tools.pingdom.com/fpt/#!/czoIzm/https://new.betterbatt.com.au/
With JS Bundling (2.1mb): http://tools.pingdom.com/fpt/#!/c8CxvG/https://new.betterbatt.com.au/

As you can see JS Bundling increases page size by 4x.

I also noticed that JS Bundling when enabled blocks the selection of role resources when adding a custom admin role.

@erikhansen
Copy link
Contributor

@erikhansen erikhansen commented Jun 6, 2016

From what I've read here and from my independent testing, it seems like the "Enable Javascript Bundling" is currently unusable and should not be recommended for production environments. The way bundling has been implemented for M2 negates much of the benefits gained by RequireJS.

I would recommend the following:

  • To prevent developers from enabling bundling without understanding its ramifications, in the next patch release, remove the "Enable Javascript Bundling" option or at least add a disclaimer beneath the option alerting the admin that the feature is not yet recommended for production.
  • Refactor the JS bundling functionality so that rather than bundling all of the JS files into a massive ~8MB file, the bundler should only bundle JS files that are known to be required by all pages. The logic behind how the bundler should know what files to include is up for discussion, but the <exclude> logic in view.xml could possibly be inverted to use an <include> logic so that only files/directories in this node would be bundled.
@hostep
Copy link
Contributor

@hostep hostep commented Jun 6, 2016

And let me just add one small addition: if your webserver is capable of running HTTP/2 (which will become more and more the case) it is really not recommended to enable bundling. Since HTTP/2 hasn't got performance problems with transferring a lot of smaller files.

@southerncomputer
Copy link
Contributor

@southerncomputer southerncomputer commented Jun 7, 2016

Expected '}'
File: bef36da44feee702d670095d789f94f7.min.js, Line: 136, Column: 49365

Getting this error on IE and Firefox plus admin section no longer functioning after enabling bundling on 2.0.6 !

This seems about as reliable as the old magento 1 compiler ! Having to load +140 .js files due to this feature not working is severely bad!

@erikhansen
Copy link
Contributor

@erikhansen erikhansen commented Jun 7, 2016

@southerncomputer Your browser will cache the JS files, so while the first page will load more slowly, subsequent pages will use the cached JS files. And since the JS is being loaded asynchronously via RequireJS, the page will render before the JS files are loaded.

That being said, it does seem a bit much to have ~118 JS files loaded for a vanilla M2 homepage—thus why I'd like to see a smarter bundling feature.

@southerncomputer
Copy link
Contributor

@southerncomputer southerncomputer commented Jun 8, 2016

#3040 this is a duplicate ?

@erikhansen
Copy link
Contributor

@erikhansen erikhansen commented Jun 8, 2016

@southerncomputer No, #3040 is not a duplicate issue of this issue. Although the two issues are somewhat related in that they deal with JS minification/bundling.

However the issue you described in your comment above might be caused by the issue reported in #3040.

@southerncomputer
Copy link
Contributor

@southerncomputer southerncomputer commented Jun 8, 2016

https://github.com/magento/magento2/blob/6ea7d2d85cded3fa0fbcf4e7aa0dcd4edbf568a6/lib/web/mage/backend/validation.js

this file throws an error in the bundle regardless if you disable jshrink or not and have bundling enabled. I will try to manually minify this file to see where the file is causing issues in bundling.

The result is in IE11/FF it will throw an unmatched } error on the developer tools console, thus rendering the big bundle unusable for the most part!

@Zaylril
Copy link

@Zaylril Zaylril commented Jun 14, 2016

I have to agree with @erikhansen in that in it's current state the bundling feature is simply not usable. Just think here for a second for a mobile user on 3G, even with the 13meg file gzipped it comes out at a whopping 3.3meg. Add this to the rest of the page and you are looking at 5meg+ for an initial page load. Not great.

Surely a much better solution would be to only bundle files which would be used on every page - or have a config where you could add files you wanted to be bundled together - whether this be automatic or not - meaning you would get some benefit from bundling files for a common.js file and then load the rest as and when they are needed.

Additionally I'm also not sure why the requireJS optimizer is not being implemented - where it combines related scripts together to build multiple bundles which are included as and when they are needed vastly reducing the number of individual files loaded, and greatly increasing performance.

Of course there is a balance to be had between file size and the number of files loaded - but as already mentioned when http2 is implemented this becomes a non-issue, however I can't really see this happening for everyone and all devices in the near future.

I think the only solution is really to create a smarter bundling feature - similar to that of the requireJS optimizer or implementing a build step where smarter bundling occurs. In this way you would reduce the number of requests down for each page and also reduce the file size as well - as was the initial intention of loaded each file separately. This really needs fixing though - especially on the checkout where there is something like 250 JS files loaded - not to mention the .html templates being loaded via AJAX too. Not the best for performance. The worst thing about this also is you can't really implement any kind of server cache on all those JS files to speed it up - as it totally relies on the browser.

@erikhansen
Copy link
Contributor

@erikhansen erikhansen commented Jun 14, 2016

@Zaylril Thanks for adding to the discussion.

@alankent I'd be curious to hear your thoughts on this issue.

@southerncomputer
Copy link
Contributor

@southerncomputer southerncomputer commented Jun 15, 2016

2.1.0-rc2 now creates 5 bundle0.js .. bundle5.js - not that is saving much space but breaks up the files to concurrently load?

@erikhansen
Copy link
Contributor

@erikhansen erikhansen commented Jun 15, 2016

@southerncomputer Thanks for reporting, but this still doesn't fix the underlying concern that someone visiting a Magento 2 homepage shouldn't be served ~8MB (uncompressed) of JS, most of which is not needed on the homepage.

@Khaleel
Copy link

@Khaleel Khaleel commented Jan 15, 2017

I agree. Webpack. Gulp. Many better tools than RequireJS there is still so much bloat in Ajax calls, phtml, knockout and then requireJS - cannot do anything until requireJS has fired every ajax and knockout call.. then mixed in with localStorage - get back to American developers

@dunarri
Copy link

@dunarri dunarri commented Feb 26, 2017

@viktor-zhuromskyy That is the size it should be, is that the result on v2.1.3?

@dunarri
Copy link

@dunarri dunarri commented Feb 26, 2017

I'm the OP (changed name) and have retested the issue on the latest Magento v2.1.5.

With the merge and bundling options turned on (also note Magento has to be in production mode to use these settings) I now get 61 requests and 1.1MB transferred on a category page for about 5s load time.

The merged js file is still kinda large at 3.4MB uncompressed and this can be seen by
ls -lh pub/static/_cache/merged/

By comparison, testing the same page with merge and bundling turned off I get 200 requests, 131 of which are JS, and 1.0MB transferred. So it would appear they have addressed the issue.
Here's the kicker though, with merge and bundling turned off my page load time was only ~3s. So it doesn't achieve the desired result of faster page loads anyway. And I'm running HTTP/1.1, I'd imagine it gets even better with HTTP/2.

Also, as I stated originally the bundle includes the large JS libraries that could be offloaded to CDN (jQuery, jQuery-UI, Knockout, RequireJS).

In conclusion, don't use the bundling feature.
If you want to improve page load, get large JS libraries from a CDN and upgrade your web server to support HTTP/2.

@pynej
Copy link

@pynej pynej commented Feb 27, 2017

The optimal setup is something like so:

HTTP/2 Nginx Proxy to Varnish Cache to Apache
With compression, minification, and etags/expiration manually set in apache as magenta doesn't so them all properly. At this point you can turn of file compilation but in reality the site really shouldn't be needing 131 separate JS @ 1MB files to build a simple page.

The only real issues is that due to how Magento does it's url rewriting rules I can't seam to get full site SSL working, and not just the admin/checkout pages.

@mombrea
Copy link

@mombrea mombrea commented Mar 9, 2017

@pynej for full site ssl, try setting both the base url and base secure url to https

php bin/magento setup:store-config:set --base-url="https://example.com/"
php bin/magento setup:store-config:set --base-url-secure="https://example.com/"
php bin/magento cache:flush

@pynej
Copy link

@pynej pynej commented Mar 9, 2017

Oh I'v tried that, The system gets really pissy and force rewrites URL's now matter how I do it when in production mode due to the HTTP Proxy layer. Oddly it seamed to work in standard mode.

@zetlen
Copy link

@zetlen zetlen commented Mar 20, 2017

@dunarri and anyone else with these unacceptably large bundles: do you still have the inline sourceMappingUrl followed by lots of base64 at the end of your bundled file?

@benheart
Copy link

@benheart benheart commented Mar 31, 2017

Hello guys, is there anyone solved this issue? I minified and merged the Javascript files and enable Javascript Bundling, but the bundled file is still very large(1M+), so home page load very slowly. I consider that can we use asynchronous js or move bundled js file to the foot of html body? But I don't know how to do that, if someone has good methods to solve this issue, please tell me, thanks very much!

@Khaleel
Copy link

@Khaleel Khaleel commented Mar 31, 2017

@hwj727 One of the things we are doing:

  • Minifiying all HTML
  • Minifying all CSS
  • Serving CSS on CDN
  • Serving Images on CDN

I am going to try serving JS on the CDN too but there is a lot of JS which has PHP inside of it - such as Knockout files etc - so not sure about that. Magento bundling is poor

To place files at the bottom will only help a small amount as it is knockout dom rendering so it will make you site look funny - this also affects data-mage-init calls.. seriously.. Magento needs to get their shit together with this Front End.. especially the cart.. no documentation on serious front end development at all..

To move JS at the bottom edit the XML files in Magento Vendor files and theme XML files

@zetlen
Copy link

@zetlen zetlen commented Apr 11, 2017

@hwj727 and @Khaleel: I'm sorry you're having so much trouble. Can I ask if either of you see the sourceMappingUrl section in your large JavaScript bundles?

@zchking
Copy link

@zchking zchking commented Apr 16, 2017

2.2M after merge Magento 2.1.6, I can accept it:)

@Khaleel
Copy link

@Khaleel Khaleel commented Apr 25, 2017

Even with bundling on Magento 2 are not using RequireJS optimiser so HTTP calls will still be over 40+ to 50+ for calling JS files (async or not async) the M2 default still has a heavy HTTP call..

@SchumacherFM
Copy link
Member

@SchumacherFM SchumacherFM commented Apr 25, 2017

Same here ... it's just a big PITA

@lingwooc
Copy link

@lingwooc lingwooc commented Jun 17, 2017

I have 2.6mb merged .min.js which isn't too bad. It doesn't gzip though which rather spoils it. I have mod_deflate setup in the magento root .htaccess and it works for other files in _cache/merged. Any fixes?

Thinking about it, why aren't static files pre-gzipped when they are deployed rather than done on the fly each time?
edit: I'm using a cdn now which resolves the gzipping issue (as well as being generally better)

@magento-engcom-team
Copy link
Contributor

@magento-engcom-team magento-engcom-team commented Sep 20, 2017

@VegasJames thank you for your report.
The fix for this issue is already fixed in 2.0.11 release

@antonkril
Copy link
Contributor

@antonkril antonkril commented Sep 26, 2017

More efficient r.js-based bundling solution: https://github.com/antonkril/magento-rjs-config

@thaddeusmt
Copy link

@thaddeusmt thaddeusmt commented Jan 30, 2018

FWIW, commenting on a closed bug... in 2.2.2 with JS bundling on (but not minification) the single JS file being output is 5.3MB right now. And our theme has bundle_size = 1MB set.

@erikhansen
Copy link
Contributor

@erikhansen erikhansen commented Aug 21, 2018

@antonkril Thanks for sharing that detailed writeup about how to optimized Magento's bundling. I look forward to going through those steps for some of the Magento sites I'm working on. Due to the work involved with creating customized bundles, I expect the most merchants/developers will not implement them.

  1. Do you anticipate a future core Magento release to ship with this type of configuration built-in?

  2. Do you know of any Gist or Github pages that share bundle configurations for a vanilla Magento 2.2.x site?

@SKovbel
Copy link

@SKovbel SKovbel commented Oct 25, 2018

2 ways

  1. Duplicate each row with .min. in view.xml, exclude section
        <item type="file">Magento_Customer::js/zxcvbn.js</item>
        <item type="file">Magento_Customer::js/zxcvbn.min.js</item>
  1. Fixing function isExcluded, ignore .min. suffix in comparasion
    if ($excludedFilePath === $filePath || $excludedFilePath === str_replace('.min.js', '.js', $filePath)) {
diff --git a/vendor/magento/module-deploy/Service/Bundle.php b/vendor/magento/module-deploy/Service/Bundle.php
index cf0d7a551..f661a69d9 100644
--- a/vendor/magento/module-deploy/Service/Bundle.php
+++ b/vendor/magento/module-deploy/Service/Bundle.php
@@ -193,7 +193,7 @@ class Bundle
         $excludedFiles = $this->bundleConfig->getExcludedFiles($area, $theme);
         foreach ($excludedFiles as $excludedFileId) {
             $excludedFilePath = $this->prepareExcludePath($excludedFileId);
-            if ($excludedFilePath === $filePath) {
+            if ($excludedFilePath === $filePath || $excludedFilePath === str_replace('.min.js', '.js', $filePath)) {
                 return true;
             }
         }
@webspeaks
Copy link

@webspeaks webspeaks commented Jan 10, 2019

For me the culprit is JS bundling. I was seeing huge JS file (minified and bundled) even in admin side. Admin was too slow. Similarly frontend was slow due to huge JS file size.

Bundling enabled:
Admin minified and bundled js size: 17.2MB

Bundling disabled:
Admin minified and bundled js size: 100kb

That's a disastrous different for me. I am using Magento 2.3.0. So I turned off js bundling (even in production mode) and life is smooth now.

Disable js bundling:
php -f bin/magento config:set dev/js/enable_js_bundling 0

@magento-engcom-team
Copy link
Contributor

@magento-engcom-team magento-engcom-team commented Sep 14, 2019

Hi @VegasJames. Thank you for your report.
The issue has been fixed in #24506 by @hostep in 2.3-develop branch
Related commit(s):

The fix will be available with the upcoming 2.3.4 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.