Async JavaScript and get rid of the all.js in the Storefront

[tobiasberge]

Problem

Currently, every JavaScript that is imported into an app's or plugin's main.js entry file and built with our composer build:js command will end up in the Storefront all.js. This is also the case for JavaScript which is added to the core Storefront.

The all.js file is static on all routes and is included (hard-coded) in the base twig template.
This is not ideal, because the shop is shipping a lot of unused JavaScript. Furthermore, with more installed apps the all.js gets bigger.

Solution

Async JS-plugin system and allow dynamic imports

The majority of the Storefront JavaScript is written in JS-plugins. Currently, the JS-plugin system is completely synchronous.
All JS-plugins are imported synchronously into the main.js entry-point before the JS-plugin registration and will therefore always end up in the all.js.
No matter if those registered JS-plugins will ever be used on a specific page of the Storefront.

Current way

// <platform-root>/src/Storefront/Resources/app/storefront/src/main.js

import CrossSellingPlugin from 'src/plugin/cross-selling/cross-selling.plugin';

// The plugin file is imported already before the plugin registraton.
PluginManager.register('CrossSelling', CrossSellingPlugin, '[data-cross-selling]');

// <platform-root>/src/Storefront/Resources/app/storefront/src/plugin-system/plugin.manager.js

class PluginManager {
    // The pluginClass is already loaded beforehand in the registry.
    _initializePlugin(pluginClass, selector, options, pluginName = false) {
        if (DomAccess.isNode(selector)) {
            return PluginManagerSingleton._initializePluginOnElement(selector, pluginClass, options, pluginName);
        }
        // ...
    }
}

New way

Instead of importing all JavaScript plugin files synchronously into the main.js, we want to offer an async approach in window.PluginManager.register().
This new async way is optional and allows it to pass an arrow function containing a dynamic import to the plugin file.
Not all JS-plugins have to be async, for example, JS-plugins which are supposed to be on every page.

After the async JS-plugin registration the initialization phase window.PluginManager.initializePlugins() checks automatically if a registered JS-plugin is async and fetches it on-demand when the plugin selector (e.g. [data-cross-selling]) is found in the current DOM. This way the current DOM determines if a JS-plugin will be loaded.

// <platform-root>/src/Storefront/Resources/app/storefront/src/main.js

// Instead of static importing the plugin file, we provide a dynamic import in order to fetch it later during "initializePlugins".
window.PluginManager.register('CrossSelling', () => import('src/plugin/cross-selling/cross-selling.plugin'), '[data-cross-selling]');

// <platform-root>/src/Storefront/Resources/app/storefront/src/plugin-system/plugin.manager.js

class PluginManager {
    // The pluginClass is fetched on-demand when the plugin selector is found in the DOM
    async _initializePluginAsync(pluginClassAsync, selector, options, pluginName = false) {
        if (DomAccess.isNode(selector)) {
            const pluginClass = (await pluginClassAsync()).default;

            return PluginManagerSingleton._initializePluginOnElement(selector, pluginClass, options, pluginName);
        }
        // ...
    }
}

Getting rid of the all.js

We want to ship all JS files from an apps dist folder <app-root>/Resources/app/storefront/dist/storefront/js/ to <platform-root>/public/theme/<theme-hash>/js/ instead of creating an all.js on the fly.
Currently, shopware only considers a single JS file within the apps /dist folder that matches the app's technical name, e.g. <app-root>/Resources/app/storefront/dist/storefront/js/my-listing-app.js.
Those dist files are collected by shopware and merged into the all.js during the theme:compile process.

Every app that has a main.js entry point will be in the all.js later.

If we want to offer the possibility to fetch JavaScript files async, we need them physically inside the <platform-root>/public/theme/<theme-hash>/js/, so the browser can actually fetch them later.

Current way

Let's look at this example with 2 installed apps:

└── custom/apps/
    ├── FancyBox/src/Resources/app/storefront/dist/storefront/js/
    │   ├── fancy-box.js      (considered by shopware)
    │   ├── async-bundle-1.js (ignored)
    │   └── async-nundle-2.js (ignored)
    └── MyListing/src/Resources/app/storefront/dist/storefront/js/
        ├── my-listing.js     (considered by shopware)
        ├── async-bundle-1.js (ignored)
        └── async-nundle-2.js (ignored)

Shopware merges all the apps JS fancy-box.js and my-listing.js into the all.js and distributes this all.js
to the <platform-root>/public/theme/<theme-hash>/js/ folder. All other JS-bundles which might have been created by Webpack are not considered.

In the theme folder, as well as in the template, we only have the all.js which looks like this:

└── public/theme/
    └── 6c7abe8363a0dfdd16929ca76c02aa35/
        ├── css/
        │   └── all.css
        └── js/
            └── all.js  

<!-- Storefront, fancy-box.js and my-listing.js are inside "all.js" -->
<script src="http://shopware.local/theme/6c7abe8363a0dfdd16929ca76c02aa35/js/all.js" defer></script>

New way

Instead of merging everything into the all.js on the fly during theme:compile, we want to keep the apps bundled JS files
and distribute them to the Sales Channel/Theme folder as they are. Each app/plugin gets a separate directory containing their JS files.

Not all JS-bundle files within the /public/theme directory will automatically generate a <script>-tag in the template.
Only the "main" JS-bundle which is named like the apps technical name (similar to the old way) will also generate a <script>-tag.
All other bundles are inside the /public/theme directory, so they can eventually be fetched later by the main bundle:

└── public/theme/
    └── 6c7abe8363a0dfdd16929ca76c02aa35/
        ├── css/
        │   └── all.css
        └── js/
            ├── storefront/
            │   ├── storefront.js     (main bundle of "storefront", generates <script>)
            │   ├── cross-selling_plugin.js
            │   └── listing_plugin.js
            ├── fancy-box/
            │   ├── fancy-box.js      (main bundle of "fancy-box", generates <script>)
            │   ├── async-bundle-1.js
            │   └── async-bundle-2.js
            └── my-listing/
                ├── my-listing.js     (main bundle of "my-listing", generates <script>)
                ├── async-bundle-1.js  
                └── async-bundle-2.js    

<!-- Storefront core, will eventually fetch "cross-selling_plugin.js" and "listing_plugin.js" -->
<script src="http://shopware.local/theme/6c7abe8363a0dfdd16929ca76c02aa35/storefront/storefront.js" defer></script>

<!-- FancyBox app, will eventually fetch "async-bundle-1.js" and "async-bundle-2.js" -->
<script src="http://shopware.local/theme/6c7abe8363a0dfdd16929ca76c02aa35/fancy-box/fancy-box.js" defer></script>

<!-- MyListing app, will eventually fetch "async-bundle-1.js" and "async-bundle-2.js" -->
<script src="http://shopware.local/theme/6c7abe8363a0dfdd16929ca76c02aa35/my-listing/my-listing.js" defer></script>

Async JS on-demand

If we distribute all bundled JavaScript from <app-root>/Resources/app/storefront/dist/storefront/js/ to <platform-root>/public/theme/<theme-hash>/js/
it would also allow dynamic imports on-demand and not only using the async JS-plugin system:

init() {
    document.querySelector('.btn-launch').addEventListener('click', async () => {
        // Only fetch needed JS after user interaction.
        const largeJS = await import('src/some-large-file');

        // do something with largeJS
    });
}

└── public/theme/
    └── 6c7abe8363a0dfdd16929ca76c02aa35/
        ├── css/
        │   └── all.css
        └── js/
            └── my-listing/
                ├── my-listing.js      (main bundle of "my-listing")
                └── some-large-file.js (created by the dynamic import)

Working example (POC)

We have created a working example of the async PluginManager outside of shopware in StackBlitz.
It contains a few async example JS-plugins. The PluginManager and it's dependencies are copied over from platform and adjusted to support async JS-plugins:

⚡ A working example in StackBlitz

Benefits

• No more growing all.js on every page/route
• No more concatenating of JS during theme:compile
• Lesser unused JavaScript
• Decreased JavaScript size overall
• Improved lighthouse audit (Reduce unused JavaScript)
• Possibility to load JS files async on-demand
• Easier debugging in case of a JS error because not everything is inside the all.js

Break strategy

We consider this as a breaking change because of several reasons:

• We cannot change existing JS-plugins to async right away. If they are modified via PluginManager.override(), the override will not take effect (without adjustments) because the JS-plugin is was not yet loaded.
• Apps/plugins need to ship more pre-build JS files and not just a single bundle dist/technical-name.js in case they are using async JS.
• In order to support dynamic imports and emitting multiple JS-bundles we will need to make changes to the Webpack config. Those changes can make pre-build JS files incompatible with older shopware versions if this were shipped right away.

New feature flag

Because of those breaks we want to make these changes opt-in via a new feature flag STOREFRONT_NEW_JS_BUNDLING.

The feature flag needs a few addtional workarounds because the current static imports inside the main.js cannot be made inside an if condition.
If we would keep the static imports, the new dynamic imports would not have an effect because the files are already imported. Therefore, we will need something like an additional main.js entry point which is used when the feature flag is active:

Default entry-point (STOREFRONT_NEW_JS_BUNDLING=0)

// <platform-root>/src/Storefront/Resources/app/storefront/src/main.js

import CrossSellingPlugin from 'src/plugin/cross-selling/cross-selling.plugin';
import ListingPlugin from 'src/plugin/listing/listing.plugin';
import OffCanvasFilterPlugin from 'src/plugin/offcanvas-filter/offcanvas-filter.plugin';
// etc...

PluginManager.register('CrossSelling', CrossSellingPlugin, '[data-cross-selling]');
PluginManager.register('Listing', ListingPlugin, '[data-listing]');
PluginManager.register('OffCanvasFilter', OffCanvasFilterPlugin, '[data-off-canvas-filter]');
// etc...

Async entry-point (STOREFRONT_NEW_JS_BUNDLING=1)

// <platform-root>/src/Storefront/Resources/app/storefront/src/main-async.js

PluginManager.register('CrossSelling', () => import('src/plugin/cross-selling/cross-selling.plugin'), '[data-cross-selling]');
PluginManager.register('Listing', () => import('src/plugin/listing/listing.plugin'), '[data-listing]');
PluginManager.register('OffCanvasFilter', () => import('src/plugin/offcanvas-filter/offcanvas-filter.plugin'), '[data-off-canvas-filter]');
// etc...

The same principle would apply to our webpack.config.js. A new config containing the needed adjustments will be used when the flag is active:

└── src/Storefront/Resources/app/storefront
    ├── webpack.config.js                   (default)
    └── webpack.multi-compiler.config.js    (STOREFRONT_NEW_JS_BUNDLING=1)

How JS-plugins are written

When writing a JS-plugin which extends from PluginBaseClass there are currently no changes required to the JS-plugin class itself. In order to make it async, it is enough to change the import to a dynamic import during PluginManager.register() and the PluginManager will load it async automatically as described above. Otherwise, the plugin will be static/global.

How JS-plugins are overwritten

This is currently one of the areas that require a change, since the plugin you are trying to override was not loaded yet.
Therefore the import of the class you are extending also needs to be async at the moment.
However, this is still under development and we are trying to find a more elegant solution for this.

// <app-root>/Resources/app/storefront/src/main.js
-import MyCustomListing from './plugin/my-custom-listing.plugin';

-window.PluginManager.override('Listing', MyCustomListing, '[data-listing]');
+window.PluginManager.override('Listing', () => import('./plugin/my-custom-listing.plugin'), '[data-listing]');

// <app-root>/Resources/app/storefront/src/my-custom-listing.plugin.js
-import Listing from 'src/plugin/listing/listing.plugin';
+const Listing = (await import('src/plugin/listing/listing.plugin')).default;

export default class MyCustomListingPlugin extends Listing {
    init() {
        super.init();

        this.addAdditionalStuff();
    }

    addAdditionalStuff() {
        this.el.insertAdjacentHTML('beforeend', '<p>Hello override</p>');
    }
}

[ascheider]

@tobiasberge

Thanks for this proposal.
Sounds good.
all.js file is really big in the storefront.
Specially if b2bsuite is used: https://issues.shopware.com/issues/B2B-998
Are there any plans to refactor b2b java script plugins also in async way?
Currently b2b has their own custom plugin system which is written in type script.

[tobiasberge]

Hi, thanks for your feedback. 👍

In general I think it would make sense that B2B (or extensions in general) would adapt to async JS-plugins if we implement this. But there are no concrete plans yet because this is still in work/proposal.

In the current structure I guess the only workaround is to load JS libraries as .min.js versions via <script> separately to the template where they are needed, because whenever you import things into the entry-point everything ends up in all.js.

The new B2B components https://www.shopware.com/en/news/shopware-b2b-components/ do not have a custom plugin system anymore and use the normal Storefront structure. The core storefront also has basic support for TypeScript now, so there is no need anymore for a custom TS setup and adding .ts files should work out of the box. However the typing experience may not be great at the moment because most files are not migrated to .ts yet.

[thesec]

I suggest to add a filter for each JS to allow each to load as needs be.

Possible filters might be

• homepage
• landingpages
• category pages
• product pages
• shop pages
• a list of specific URLs

[tobiasberge]

Hi, thanks for the feedback. Sorry if this is a bit long 😆

We were also thinking about this and then we came up with the async PluginManager solution.
I assume you mean that there should be already different JS <script> for different pages? And we would have different entry points like "global", "detail-page", "homepage" etc. which results in different JS in those pages/areas?

The problem we found with "splitting" the Storefront JS into different "areas" like "landingpage", "category page" etc. was that we don't always know which JavaScript will be really needed on a page. Especially CMS pages.

Let's imagine for example we have an JS entry point for "landingpage" and we will put all the JS into it which is needed for a landingpage. But now the user is adding stuff like "Image Gallery", "Product Slider", "Product Buy Box" etc. Those CMS elements need specific JS in order to work.

So I guess we would have to either always add the all the JS for components like "Image Gallery" into all areas which could theoretically render such a CMS element. For example the "landingpage" JavaScript always has the JS for "Image Gallery", "Product Slider" etc. just in case it will be rendered in the HTML via the CMS. But then we would also end up with a lot of unused JS if those elements are never used. If we would do it in a "filtered" way, we would maybe have to put the filtering one level down to the CMS elements or something.

And therefore we thought, why not let the actual DOM decide which JS will be loaded in the end? Then it would be completely dynamic and when there is a new element added to the page which has a JS-plugin selector inside (e.g. data-image-gallery="true"), the JS-plugin will be fetched on-demand by the PluginManager. And we do not have to deal with deciding "which JS is on which page".

However, I also see the benefit of having JS really only on specific pages. For example checkout. The current solution has the downside that it only works with JS-plugins and the PluginManager out of the box. (Or by doing manual dynamic imports) You always have some "wrapper" JavaScript where all the Plugins are registered still on every page. But I think this is negligible compared to the actual JS-plugins containing all the code/logic.

Maybe this could be an addition to the current system. But then you would have introduce some form of additional configuration to offer different entry points for the different areas and probably most of the JS would have been written as Plugins and being fetched async anyway. Where would you like to see such a filter/configuration? Really curious to hear your thoughts on this.

[thesec]

I was actually thinking of adding the filter on e.g. a per-plugin basis as well as for cms-pages and have the admin decide on the setting within the administration.

That way the default would persist for all things SWAG and other can be fine-tuned as demands go.
For the remainder the pool to choose from would already be narrowed down, speeding up the rendering.

[jacobseated]

I think this is a very sensible and logical thing to implement, and it should probably also be done for CSS; it makes no sense to have an all.js or even all.css for that matter. It would have been much better to simply include the code as local script and style blocks on pages where it's actually used.

Currently we are some devs who are trying to hack around it. E.g. We use Puppeteer at my workplace to create a "critical" css file that we load in addition to the existing CSS. This is slower overall, and I greatly dislike it, but at least it helps load the critical elements faster without too much shifting. We have also had talks about eliminating Shopware's own JavaScript entirely because it is so poorly optimized, but it seems like a very difficult thing to do without creating your own Storefront.

Currently we are in the process of creating our own tailwind based theme to avoid Shopware's bloated CSS, but the JavaScript may be more difficult to avoid without recreating the entire JS as well. Because of this suggestion, I think we will wait and see before deciding on what to do about the JS.

[jacobseated]

For critical CSS, this is what we use: https://github.com/jesperingels/shopware-critical-css
One of the only modifications we made was to have our theme automatically load the generated critical .css file for the given page. Basically it just looks if a file exists from within our child theme twig, and if it does, includes it. Unfortunately also conflicts with caching, so we have to clear:cache after generating the critical CSS. It's just a minor inconvenience.

The disadvantage is we need to add each page with a known "unique" above-the-fold layout to the critical CSS .json to have it created properly. But, for the most part we have just used a generalised file anyway. E.g. One for category, one for products, and one for the front page.

I even tested removing the duplicated/unused CSS, and it somewhat worked, with minor errors that would probably have been easy to fix. Now, this is totally a hack and not something I'd recommend as a long term solution. Long-term, I'd make my own Storefront theme and avoid this whole mess, which is what we are doing now.

It shows that something is doable with existing tools.

Another issue is, Shopware is kinda abusers of div-it-is; there is way too many elements per-page. We are also trying to get rid of some of that.

Done properly, there should be some code to check whether a given CMS element is used on a page, and if so, include the relevant CSS/JS only once. You could even have something that checks for "commonly used elements", and then only include those commonly used elements in a type of all.css, while the rest gets loaded via individual .css files the few places where they might be used. This does increase the number of HTTP requests, although that will not matter as much for HTTP/2 or HTTP/3 – but someone that works on this needs to test how to best go about it.

Ideally, I'd aim to push the "main" CSS to 14kB (compressed and minified), but that's probably not anywhere near realistic with so much stuff.

[thesec]

"div-it-is" ... 🤣 ... totally nailed it!

Best example: 50+% of the "cms"'s layers can be removed without any display issues at all happening in the storefront.

Finally learning Bootstrap might be a good starter for Schöppingen ... but getting OT ;)

Personally, I am having very good results with a toolchain of purgecss & cleancss followed by a px2rem conversion.

Back to "OT" this thread might make it into a standalone discussion on CSS cleanup?

[luukskeur]

FYI, whoever is looking for optimizing the Shopware-overridden-Bootstrap CSS, I used the "StorefrontBootstrap" version in my theme.json which is a way more cleaner version of the default storefront and doesn't come with as much overhead. I don't have the exact numbers but it saved me about ±50% of the actual filesize and a lot cleaner code.

"style": [
    "app/storefront/src/scss/overrides.scss",
    "@StorefrontBootstrap",
    "app/storefront/src/scss/base.scss"
],

[jacobseated]

Just an update on what we ended up doing :-)

We ended up ditching the critical css fix around December/January, and instead making our own Tailwind+Flowbite based theme. Shopware could totally do something similar in their default Themes and allow all Shopware users to benefit.

Considering the amount of development that goes into this sort of thing, and number of unforseen problems, I would personally not rush it the way we did. That was a mistake. It was very tough work, with probably a few all-nighters in my own case.

Also, I'd really want to take a skeptical approach to any suggestion that we absolutely must reach a 100. That's not what Google themselves advices, and they have actually downplayed the significance this score has on ranking (it is still a ranking factor). Unfurtunately, I don't think people have a realistic feel of what will actually be the end-result, and they end up becoming obsessed with these online tools, (search console included), and it's just dissapointing when you then don't see meaningful results.

E.g. Even after reaching between 80-100 pagespeed score, we still have not noticed significant improvements. Our average positions are not significantly higher imo, and any gains can just as well be attributed to other improvements, including improving our content and website design. This is very similar to what I experienced on my personal blog after reaching 100.

There are other things that's way more important, mainly content, usability, and design. But, that is not saying you should not do it if you get the oppertunity.

[seakrebel]

We ended up ditching the critical css fix around December/January, and instead making our own Tailwind+Flowbite based theme. Shopware could totally do something similar in their default Themes and allow all Shopware users to benefit.

Was it hard to create own Tailwind+Flowbite based theme? Is there any skeleton you could provide for a quick start? I've lost so much time trying to clean the shopware styling in the child theme of the default theme

[sr-mothership]

Would it also be possible to get rid of Webpack and use Vite instead or would this delay fixing the core issue too much?

[kingschnulli]

We are currently at this also, for critical css the only viable solution seems to generate the critical css using penthouse. But this has a major drawback as penthouse uses puppeteer under the hood - and this leads to a lot of trouble with missing libraries or obscure errors on the ci/deployment server.

For now we use our own little custom server that accepts an URL and generates the critical css on demand, sending it back to the SW installation. Critical CSS is generated by using current route + salesChannelId as key then stored to cache.

All of this is a weird drag but works - I would really love to see some solution which eliminates all of that.

Also splitting js is the last corner piece for 100% on pagespeed scores.

[luukskeur]

Included in the v.6.6 RC: https://www.shopware.com/en/news/shopware-6-6-rc/#66-rc-storefront-changes 🔥

[ROBJkE]

How about mention it in the official documentation....

[tobiasberge]

It was added to the docs under the v6.6 (RC) toggle: https://developer.shopware.com/docs/v6.6rc/guides/plugins/plugins/storefront/add-custom-javascript.html#registering-an-async-plugin