New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance with many files #872

Open
bernhardw opened this Issue Jun 24, 2015 · 183 comments

Comments

Projects
None yet
@bernhardw

bernhardw commented Jun 24, 2015

Hello,

Thanks a lot for all the great work on jspm 馃憤

During development, when importing libraries split across many files such as React or Angular2, es6-module-loader loads all the required files individually which takes around 2-4 seconds. It helps to import a provided build instead, but not all packages include one.

Is there a suggested way around it?

@guybedford

This comment has been minimized.

Show comment
Hide comment
@guybedford

guybedford Jun 24, 2015

Member

There are a number of approaches here, but no prescribed workflow currently.

In an ideal world the development server would be a fast HTTP/2 server with incremental caching understanding SystemJS and doing server-side transpilation and providing hot reloading.

Note that just using a better HTTP server for development can make a lot of difference. Some of them are very badly optimized for large numbers of requests.

@geelen is doing some amazing work on these sort of ideas in http://npmjs.org/package/jspm-server, but I don't know how ready it is.

While we wait for the future, the best workflow is simply to have common bundles that you load at the top of your app:

  jspm bundle app/**/* - [app/**/*] dependency-bundle.js

You then include dependency-bundle.js at the top of the page, and you'll only be loading your own application code as separate files in the browser.

Other users have quite advanced pre-compilation gulp tasks to do ES6 transpilation locally before loading separate files in the browser.

It's an area where users are still on there own for now, so please share what you learn with others as we collectively refine these workflows.

Member

guybedford commented Jun 24, 2015

There are a number of approaches here, but no prescribed workflow currently.

In an ideal world the development server would be a fast HTTP/2 server with incremental caching understanding SystemJS and doing server-side transpilation and providing hot reloading.

Note that just using a better HTTP server for development can make a lot of difference. Some of them are very badly optimized for large numbers of requests.

@geelen is doing some amazing work on these sort of ideas in http://npmjs.org/package/jspm-server, but I don't know how ready it is.

While we wait for the future, the best workflow is simply to have common bundles that you load at the top of your app:

  jspm bundle app/**/* - [app/**/*] dependency-bundle.js

You then include dependency-bundle.js at the top of the page, and you'll only be loading your own application code as separate files in the browser.

Other users have quite advanced pre-compilation gulp tasks to do ES6 transpilation locally before loading separate files in the browser.

It's an area where users are still on there own for now, so please share what you learn with others as we collectively refine these workflows.

@bernhardw

This comment has been minimized.

Show comment
Hide comment
@bernhardw

bernhardw Jun 25, 2015

Thank you Guy for the detailed answer. I was hoping to get rid of my pre-compilation tasks during development, but creating a dependency bundle works for now.

And with --inject, you don't even need to include your script manually:

jspm bundle app/**/* - [app/**/*] dependency-bundle.js --inject

bernhardw commented Jun 25, 2015

Thank you Guy for the detailed answer. I was hoping to get rid of my pre-compilation tasks during development, but creating a dependency bundle works for now.

And with --inject, you don't even need to include your script manually:

jspm bundle app/**/* - [app/**/*] dependency-bundle.js --inject
@geelen

This comment has been minimized.

Show comment
Hide comment
@geelen

geelen Jun 26, 2015

jspm-server is definitely usable right now, and I'd love feedback on whether it improves your workflow. Basically, you can mark files as being "safe" to live-reload, and if those files change it won't have to reload the whole browser. It's still a work in progress, and will definitely benefit from changes coming down in the next version of JSPM & SystemJS, but right now I'm using it for all of my projects.

That said, doing the dependency bundling will definitely help as well, so that's a neat trick. I'd love there to be a higher-level command for doing that, something like jspm bundle-deps that bundles everything in jspm_packages? Would that be possible?

Also, @guybedford, you mention "better HTTP servers". Do you know of any? I want to find the best and build jspm-server around it. I don't know anything that supports HTTP/2 either. I have an issue for it here: geelen/jspm-server#4

geelen commented Jun 26, 2015

jspm-server is definitely usable right now, and I'd love feedback on whether it improves your workflow. Basically, you can mark files as being "safe" to live-reload, and if those files change it won't have to reload the whole browser. It's still a work in progress, and will definitely benefit from changes coming down in the next version of JSPM & SystemJS, but right now I'm using it for all of my projects.

That said, doing the dependency bundling will definitely help as well, so that's a neat trick. I'd love there to be a higher-level command for doing that, something like jspm bundle-deps that bundles everything in jspm_packages? Would that be possible?

Also, @guybedford, you mention "better HTTP servers". Do you know of any? I want to find the best and build jspm-server around it. I don't know anything that supports HTTP/2 either. I have an issue for it here: geelen/jspm-server#4

@OrKoN

This comment has been minimized.

Show comment
Hide comment
@OrKoN

OrKoN Jun 27, 2015

Contributor

@geelen I am using nginx with spdy support here: https://github.com/OrKoN/jspm-ember-playground It's faster than any node-based servers I used. But I guess it would be hard to package nginx into the jspm-server.

Also there is node-spdy server https://github.com/indutny/node-spdy and https://github.com/molnarg/node-http2 which is not fully working right now (see molnarg/node-http2#114).

Contributor

OrKoN commented Jun 27, 2015

@geelen I am using nginx with spdy support here: https://github.com/OrKoN/jspm-ember-playground It's faster than any node-based servers I used. But I guess it would be hard to package nginx into the jspm-server.

Also there is node-spdy server https://github.com/indutny/node-spdy and https://github.com/molnarg/node-http2 which is not fully working right now (see molnarg/node-http2#114).

@OrKoN

This comment has been minimized.

Show comment
Hide comment
@OrKoN

OrKoN Jun 27, 2015

Contributor

I have also written a blog post about optimizing the performance with many files using gulp and nginx https://60devs.com/optimizing--default-jspm-workflow-with-gulp-and-nginx.html

@geelen I wonder if jspm-server can replace the gulp stuff I have. Is there a tutorial on how to migrate an app to jspm-server? Is there gulp-jspm-server or smth similar?

Contributor

OrKoN commented Jun 27, 2015

I have also written a blog post about optimizing the performance with many files using gulp and nginx https://60devs.com/optimizing--default-jspm-workflow-with-gulp-and-nginx.html

@geelen I wonder if jspm-server can replace the gulp stuff I have. Is there a tutorial on how to migrate an app to jspm-server? Is there gulp-jspm-server or smth similar?

@geelen

This comment has been minimized.

Show comment
Hide comment
@geelen

geelen Jun 29, 2015

There's not really, jspm-server is sort of designed to be run independently of gulp or similar things.

I'd love to get a version of nginx statically linked and shipped inside an NPM package 馃槂 I wonder if that's possible...

geelen commented Jun 29, 2015

There's not really, jspm-server is sort of designed to be run independently of gulp or similar things.

I'd love to get a version of nginx statically linked and shipped inside an NPM package 馃槂 I wonder if that's possible...

@OrKoN

This comment has been minimized.

Show comment
Hide comment
@OrKoN

OrKoN Jun 30, 2015

Contributor

@geelen I think this should be possible similar to this https://jamie.curle.io/posts/webfaction-installing-custom-nginx/ But I guess it needs to be compiled on npm install.

Contributor

OrKoN commented Jun 30, 2015

@geelen I think this should be possible similar to this https://jamie.curle.io/posts/webfaction-installing-custom-nginx/ But I guess it needs to be compiled on npm install.

@bernhardw

This comment has been minimized.

Show comment
Hide comment
@bernhardw

bernhardw Jun 30, 2015

I've set up a simple static file server based on http2 to play around, but I see no difference in performance between http2 and http. It doesn't seem to multiplex, but HTTP/2 is used. Any ideas why?

bernhardw commented Jun 30, 2015

I've set up a simple static file server based on http2 to play around, but I see no difference in performance between http2 and http. It doesn't seem to multiplex, but HTTP/2 is used. Any ideas why?

@jpray

This comment has been minimized.

Show comment
Hide comment
@jpray

jpray Jul 4, 2015

Regarding solutions other than HTTP/2... it seems that if you want even one unbundled ES6 module to be loaded while developing, then babel is what is affecting the load time because it dynamically loading dozens and dozens of dependencies. @guybedford, is there a way we can bundle the modules for babel into a single file?

Update: Tried it out and yes, babel can be bundled.

jspm bundle babel .tmp/js/bundles/babel.js

---index.html---

  <script src='/jspm_packages/es6-module-loader.js'></script> <!-- optional: to load ES6-style modules -->
  <script src='/jspm_packages/system.js'></script><!-- required -->
  <script src='/config.js'></script><!-- required -->
  <script src='/.tmp/js/bundles/babel.js'></script>

jpray commented Jul 4, 2015

Regarding solutions other than HTTP/2... it seems that if you want even one unbundled ES6 module to be loaded while developing, then babel is what is affecting the load time because it dynamically loading dozens and dozens of dependencies. @guybedford, is there a way we can bundle the modules for babel into a single file?

Update: Tried it out and yes, babel can be bundled.

jspm bundle babel .tmp/js/bundles/babel.js

---index.html---

  <script src='/jspm_packages/es6-module-loader.js'></script> <!-- optional: to load ES6-style modules -->
  <script src='/jspm_packages/system.js'></script><!-- required -->
  <script src='/config.js'></script><!-- required -->
  <script src='/.tmp/js/bundles/babel.js'></script>
@OrKoN

This comment has been minimized.

Show comment
Hide comment
@OrKoN

OrKoN Jul 4, 2015

Contributor

@bernhardw According to my tests there is no difference too. I guess http/2 may be helpful in production or if the server leverages the server push technologies. I have tried http2 module you mention but it didn't work with Chrome as a client. How did you get it work?

Contributor

OrKoN commented Jul 4, 2015

@bernhardw According to my tests there is no difference too. I guess http/2 may be helpful in production or if the server leverages the server push technologies. I have tried http2 module you mention but it didn't work with Chrome as a client. How did you get it work?

@bernhardw

This comment has been minimized.

Show comment
Hide comment
@bernhardw

bernhardw Jul 5, 2015

@OrKoN I also couldn't get serve-static to work, but it did work with the less used node-static.

var fs = require('fs');
var http2 = require('http2');
var fileServer = require('node-static');
var file = new fileServer.Server('./public');

var options = {
    key: fs.readFileSync('./localhost.key'),
    cert: fs.readFileSync('./localhost.crt')
};

var server = http2.createServer(options, function (req, res) {
    req.addListener('end', function () {
        file.serve(req, res);
    }).resume();
});

server.listen(8080);

bernhardw commented Jul 5, 2015

@OrKoN I also couldn't get serve-static to work, but it did work with the less used node-static.

var fs = require('fs');
var http2 = require('http2');
var fileServer = require('node-static');
var file = new fileServer.Server('./public');

var options = {
    key: fs.readFileSync('./localhost.key'),
    cert: fs.readFileSync('./localhost.crt')
};

var server = http2.createServer(options, function (req, res) {
    req.addListener('end', function () {
        file.serve(req, res);
    }).resume();
});

server.listen(8080);
@bernhardw

This comment has been minimized.

Show comment
Hide comment
@bernhardw

bernhardw Jul 5, 2015

@jpray Thanks, that did bring it down from ~1200ms to ~750ms. For anyone else trying to bundle the babel runtime, you can and should disable the runtime in your config.js:

System.config({
  "baseURL": "/",
  "transpiler": "babel",
  "babelOptions": {
    // "optional": [
    //   "runtime"
    // ],
    "blacklist": []
  },
  "paths": {
    "*": "*.js",
    "github:*": "jspm_packages/github/*.js",
    "npm:*": "jspm_packages/npm/*.js"
  }
});

That seems to be all there is to shave off. These are my network requests now:

network

bernhardw commented Jul 5, 2015

@jpray Thanks, that did bring it down from ~1200ms to ~750ms. For anyone else trying to bundle the babel runtime, you can and should disable the runtime in your config.js:

System.config({
  "baseURL": "/",
  "transpiler": "babel",
  "babelOptions": {
    // "optional": [
    //   "runtime"
    // ],
    "blacklist": []
  },
  "paths": {
    "*": "*.js",
    "github:*": "jspm_packages/github/*.js",
    "npm:*": "jspm_packages/npm/*.js"
  }
});

That seems to be all there is to shave off. These are my network requests now:

network

@guybedford

This comment has been minimized.

Show comment
Hide comment
@guybedford

guybedford Jul 8, 2015

Member

Two perf tips along these lines, and why this is hard currently:

  1. Finding a performant HTTP/2 server that does in-memory caching of files is hard. If you know any, please share :)
  2. When serving jspm_packages, it should actually be served with a far-future expires header so that it is cached in the browser and never requested again. That way just the local application folders are refreshed. Again, knowing to set this and how to set it is not obvious when running a dev server.
Member

guybedford commented Jul 8, 2015

Two perf tips along these lines, and why this is hard currently:

  1. Finding a performant HTTP/2 server that does in-memory caching of files is hard. If you know any, please share :)
  2. When serving jspm_packages, it should actually be served with a far-future expires header so that it is cached in the browser and never requested again. That way just the local application folders are refreshed. Again, knowing to set this and how to set it is not obvious when running a dev server.
@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 8, 2015

Contributor

The caching is really complicated though cause in dev mode I currently have to constantly flush the cache (using a nice chrome extension i found for single click expiration) because my .map files are heavily cached and out of date all the time. Any ideas on that one?

Contributor

lookfirst commented Jul 8, 2015

The caching is really complicated though cause in dev mode I currently have to constantly flush the cache (using a nice chrome extension i found for single click expiration) because my .map files are heavily cached and out of date all the time. Any ideas on that one?

@guybedford

This comment has been minimized.

Show comment
Hide comment
@guybedford

guybedford Jul 8, 2015

Member

@lookfirst only cache jspm_packages, not anything else (the code you write yourself). Because paths in jspm_packages are versioned, updates get forced. Linking workflows break down here, and need custom rules though.

Member

guybedford commented Jul 8, 2015

@lookfirst only cache jspm_packages, not anything else (the code you write yourself). Because paths in jspm_packages are versioned, updates get forced. Linking workflows break down here, and need custom rules though.

@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 8, 2015

Contributor

@guybedford Right, but my point is that right now, I'm hard flushing the entire chrome cache every time I reload the browser in order to get .map files to work properly. That would kill any benefit of caching jspm_packages.

Contributor

lookfirst commented Jul 8, 2015

@guybedford Right, but my point is that right now, I'm hard flushing the entire chrome cache every time I reload the browser in order to get .map files to work properly. That would kill any benefit of caching jspm_packages.

@guybedford

This comment has been minimized.

Show comment
Hide comment
@guybedford

guybedford Jul 8, 2015

Member

I guess a no-cache header on the .map files?

Member

guybedford commented Jul 8, 2015

I guess a no-cache header on the .map files?

@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 8, 2015

Contributor

Great idea! =) Lol... (feeling dumb, i should have thought of that)...

On Wed, Jul 8, 2015 at 10:53 AM, Guy Bedford notifications@github.com
wrote:

I guess a no-cache header on the .map files?


Reply to this email directly or view it on GitHub
#872 (comment).

Contributor

lookfirst commented Jul 8, 2015

Great idea! =) Lol... (feeling dumb, i should have thought of that)...

On Wed, Jul 8, 2015 at 10:53 AM, Guy Bedford notifications@github.com
wrote:

I guess a no-cache header on the .map files?


Reply to this email directly or view it on GitHub
#872 (comment).

@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 8, 2015

Contributor

Sad face. BrowserSync/browser-sync#719

On Wed, Jul 8, 2015 at 11:38 AM, Jon Stevens latchkey@gmail.com wrote:

Great idea! =) Lol... (feeling dumb, i should have thought of that)...

On Wed, Jul 8, 2015 at 10:53 AM, Guy Bedford notifications@github.com
wrote:

I guess a no-cache header on the .map files?


Reply to this email directly or view it on GitHub
#872 (comment).

Contributor

lookfirst commented Jul 8, 2015

Sad face. BrowserSync/browser-sync#719

On Wed, Jul 8, 2015 at 11:38 AM, Jon Stevens latchkey@gmail.com wrote:

Great idea! =) Lol... (feeling dumb, i should have thought of that)...

On Wed, Jul 8, 2015 at 10:53 AM, Guy Bedford notifications@github.com
wrote:

I guess a no-cache header on the .map files?


Reply to this email directly or view it on GitHub
#872 (comment).

@geelen

This comment has been minimized.

Show comment
Hide comment
@geelen

geelen Jul 8, 2015

@lookfirst have you looked into jspm-server? We should be able to add some caching rules in there (always cache jspm_packages, never cache .map files) pretty easily, no?

geelen commented Jul 8, 2015

@lookfirst have you looked into jspm-server? We should be able to add some caching rules in there (always cache jspm_packages, never cache .map files) pretty easily, no?

@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 8, 2015

Contributor

I'd rather use browsersync as the basis. I wish your server just extended
that. I also don't really see the value proposition of jspm-server over
what I've already got setup.

On Wed, Jul 8, 2015 at 3:38 PM, Glen Maddern notifications@github.com
wrote:

@lookfirst https://github.com/lookfirst have you looked into jspm-server
https://github.com/geelen/jspm-server? We should be able to add some
caching rules in there (always cache jspm_packages, never cache .map
files) pretty easily, no?


Reply to this email directly or view it on GitHub
#872 (comment).

Contributor

lookfirst commented Jul 8, 2015

I'd rather use browsersync as the basis. I wish your server just extended
that. I also don't really see the value proposition of jspm-server over
what I've already got setup.

On Wed, Jul 8, 2015 at 3:38 PM, Glen Maddern notifications@github.com
wrote:

@lookfirst https://github.com/lookfirst have you looked into jspm-server
https://github.com/geelen/jspm-server? We should be able to add some
caching rules in there (always cache jspm_packages, never cache .map
files) pretty easily, no?


Reply to this email directly or view it on GitHub
#872 (comment).

@geelen

This comment has been minimized.

Show comment
Hide comment
@geelen

geelen Jul 8, 2015

Well, it's aware of the dependencies within your SystemJS build, so it's able to live-reload modules within your application without having to reload the browser, which will always be quicker. But basically, I want a development server for JSPM that's as fast as possible out the box, without requiring any configuration (beyond adding some export let __hotReload = true statements around the place)

I had hoped I could port jspm-server to be an extension of BrowserSync but I think that might be too much work for not much gain at this point.

geelen commented Jul 8, 2015

Well, it's aware of the dependencies within your SystemJS build, so it's able to live-reload modules within your application without having to reload the browser, which will always be quicker. But basically, I want a development server for JSPM that's as fast as possible out the box, without requiring any configuration (beyond adding some export let __hotReload = true statements around the place)

I had hoped I could port jspm-server to be an extension of BrowserSync but I think that might be too much work for not much gain at this point.

@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 8, 2015

Contributor

The gain is the functionality of browsersync without having to do much
work. If your code could live within the context of a middleware, that
would be optimal.

I'd worry the live reload of modules would break angular, no? It also seems
like there could be potential for stuff to get messy there. I'd hate to
have to debug a reload failure. Also adding __hotReload to all my files
isn't very ideal.

So far, what I've got setup is pretty darn easy and fast. Gulp manages the
watching of files and only recompiles ones (es6/less/templates) that
change. I use my gulp-helpers so I don't have to rewrite gulp files ever
time I start a new project.

https://github.com/lookfirst/gulp-helpers

Now, just waiting to see if browsersync can set the headers correctly in a
middleware and I'll be all set.

Thanks for piping in... I did look at jspm-server, but it just didn't
scratch my itch enough. =)

On Wed, Jul 8, 2015 at 4:25 PM, Glen Maddern notifications@github.com
wrote:

Well, it's aware of the dependencies within your SystemJS build, so it's
able to live-reload modules within your application without having to
reload the browser, which will always be quicker. But basically, I want a
development server for JSPM that's as fast as possible out the box, without
requiring any configuration (beyond adding some export let __hotReload =
true statements around the place)

I had hoped I could port jspm-server to be an extension of BrowserSync but
I think that might be too much work for not much gain at this point.


Reply to this email directly or view it on GitHub
#872 (comment).

Contributor

lookfirst commented Jul 8, 2015

The gain is the functionality of browsersync without having to do much
work. If your code could live within the context of a middleware, that
would be optimal.

I'd worry the live reload of modules would break angular, no? It also seems
like there could be potential for stuff to get messy there. I'd hate to
have to debug a reload failure. Also adding __hotReload to all my files
isn't very ideal.

So far, what I've got setup is pretty darn easy and fast. Gulp manages the
watching of files and only recompiles ones (es6/less/templates) that
change. I use my gulp-helpers so I don't have to rewrite gulp files ever
time I start a new project.

https://github.com/lookfirst/gulp-helpers

Now, just waiting to see if browsersync can set the headers correctly in a
middleware and I'll be all set.

Thanks for piping in... I did look at jspm-server, but it just didn't
scratch my itch enough. =)

On Wed, Jul 8, 2015 at 4:25 PM, Glen Maddern notifications@github.com
wrote:

Well, it's aware of the dependencies within your SystemJS build, so it's
able to live-reload modules within your application without having to
reload the browser, which will always be quicker. But basically, I want a
development server for JSPM that's as fast as possible out the box, without
requiring any configuration (beyond adding some export let __hotReload =
true statements around the place)

I had hoped I could port jspm-server to be an extension of BrowserSync but
I think that might be too much work for not much gain at this point.


Reply to this email directly or view it on GitHub
#872 (comment).

@geelen

This comment has been minimized.

Show comment
Hide comment
@geelen

geelen Jul 11, 2015

Yeah, angular makes it tough. I don't know exactly how to handle reloading there. And even with jspm-server, there's always times you have to fully reload, and the speed is still an issue.

@guybedford back to the original point. Could we do something silly like run SystemJS in a serviceworker? Have it manage transpilation & caching on the client side, not the server? It seems like a good fit to me, but maybe I'm missing something...

geelen commented Jul 11, 2015

Yeah, angular makes it tough. I don't know exactly how to handle reloading there. And even with jspm-server, there's always times you have to fully reload, and the speed is still an issue.

@guybedford back to the original point. Could we do something silly like run SystemJS in a serviceworker? Have it manage transpilation & caching on the client side, not the server? It seems like a good fit to me, but maybe I'm missing something...

@guybedford

This comment has been minimized.

Show comment
Hide comment
@guybedford

guybedford Jul 11, 2015

Member

@geelen my previous plan was to build a jspm server that did the plugin transpilation and es6 transpilation. Basically it would run the loader pipeline on the server first, and just send the ready sources to the client. Then you get browser-cached transpiled sources, and it's a solid foundation development server to use with the native loader too in future.

Member

guybedford commented Jul 11, 2015

@geelen my previous plan was to build a jspm server that did the plugin transpilation and es6 transpilation. Basically it would run the loader pipeline on the server first, and just send the ready sources to the client. Then you get browser-cached transpiled sources, and it's a solid foundation development server to use with the native loader too in future.

@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 11, 2015

Contributor

@guybedford I thought about that as well and in practice, it was just easier to setup gulp with browsersync. If running the loader pipeline on the server first means also getting a single file to the server, then that doesn't work well for me cause it makes it harder to debug things. The individual files are nice during development so that I can quickly see which file has issues and go to that in my IDE.

Contributor

lookfirst commented Jul 11, 2015

@guybedford I thought about that as well and in practice, it was just easier to setup gulp with browsersync. If running the loader pipeline on the server first means also getting a single file to the server, then that doesn't work well for me cause it makes it harder to debug things. The individual files are nice during development so that I can quickly see which file has issues and go to that in my IDE.

@geelen

This comment has been minimized.

Show comment
Hide comment
@geelen

geelen Jul 13, 2015

So the native loader won't do any transpilation?

That feels like a use-case that's already covered pretty well but the webpack dev server, and there are plenty of issues with it. Reloading (they call it Hot Module Replacement) is pretty complex when you have to serialise a chunk down to the browser, in particular. When the loader knows it's running in the browser (except in bundle mode of course) it can be a bit smarter, in my mind.

That's why I've thought in-browser caching would be the way forward. But you're right, a serviceworker is disconnected enough from the browser context that it'd be the same problems as I just mentioned. So maybe just adding some caching of transpiled sources into SystemJS (and in development only) is the simpler approach?

geelen commented Jul 13, 2015

So the native loader won't do any transpilation?

That feels like a use-case that's already covered pretty well but the webpack dev server, and there are plenty of issues with it. Reloading (they call it Hot Module Replacement) is pretty complex when you have to serialise a chunk down to the browser, in particular. When the loader knows it's running in the browser (except in bundle mode of course) it can be a bit smarter, in my mind.

That's why I've thought in-browser caching would be the way forward. But you're right, a serviceworker is disconnected enough from the browser context that it'd be the same problems as I just mentioned. So maybe just adding some caching of transpiled sources into SystemJS (and in development only) is the simpler approach?

@guybedford

This comment has been minimized.

Show comment
Hide comment
@guybedford

guybedford Jul 13, 2015

Member

@geelen the reason I liked the server though was that it would be equally useful in development and production. I do still plan to work on this over the coming months, but it will be a little while unfortunately. With browser caching we have a development-only optimization, that may be confused for a production workflow.

The above server is also the natural home for hot reloading, so would be nice to merge efforts. What about the serializing gets complex there? You have been working on this problem far more than me though, so interested to hear more from your experiences.

Also in terms of the future of hot-reloading, had an interesting chat with @paulirish, where he described a workflow of editing ES6 in the Chrome devtools and having that hot-reload. V8 optimizations can do a lot more there than any other approach (code updates without re-execution) so it seems like this is where things should go for hot-reloading eventually. This workflow would then (in some distant future where V8 supports modules) be based on the loader listening to the edit event of a transpiled module from devtools, and then sending back the transpilation to be hot-swapped without refreshing the page. Food for thought :)

Member

guybedford commented Jul 13, 2015

@geelen the reason I liked the server though was that it would be equally useful in development and production. I do still plan to work on this over the coming months, but it will be a little while unfortunately. With browser caching we have a development-only optimization, that may be confused for a production workflow.

The above server is also the natural home for hot reloading, so would be nice to merge efforts. What about the serializing gets complex there? You have been working on this problem far more than me though, so interested to hear more from your experiences.

Also in terms of the future of hot-reloading, had an interesting chat with @paulirish, where he described a workflow of editing ES6 in the Chrome devtools and having that hot-reload. V8 optimizations can do a lot more there than any other approach (code updates without re-execution) so it seems like this is where things should go for hot-reloading eventually. This workflow would then (in some distant future where V8 supports modules) be based on the loader listening to the edit event of a transpiled module from devtools, and then sending back the transpilation to be hot-swapped without refreshing the page. Food for thought :)

@lookfirst

This comment has been minimized.

Show comment
Hide comment
@lookfirst

lookfirst Jul 13, 2015

Contributor

Well, this should help... BrowserSync/browser-sync#719

Contributor

lookfirst commented Jul 13, 2015

Well, this should help... BrowserSync/browser-sync#719

@kdex

This comment has been minimized.

Show comment
Hide comment
@kdex

kdex Apr 11, 2016

@mgamsjager I'm still measuring around 1200 ms just to resolve System.import("app") when DevTools is completely closed. Shouldn't this be more immediate, given that the bundle has already been consumed at this point?

kdex commented Apr 11, 2016

@mgamsjager I'm still measuring around 1200 ms just to resolve System.import("app") when DevTools is completely closed. Shouldn't this be more immediate, given that the bundle has already been consumed at this point?

@jackfranklin

This comment has been minimized.

Show comment
Hide comment
@jackfranklin

jackfranklin Apr 11, 2016

Contributor

I might suggest opening a separate issue for this? Purely because this thread is quite long, it's hard to track. Might be easier to debug in separate issue + with a repository that the issue can be reproduced on :)

Contributor

jackfranklin commented Apr 11, 2016

I might suggest opening a separate issue for this? Purely because this thread is quite long, it's hard to track. Might be easier to debug in separate issue + with a repository that the issue can be reproduced on :)

@zakdances

This comment has been minimized.

Show comment
Hide comment
@zakdances

zakdances Apr 11, 2016

Yeah, I'll see if I can make a test case for this. Then I'll make a separate issue?

I think this problem is caused by having your source files in a different directory then jspm.browser.js.

My theory: When you bundle, inject, watch, then launch your app in the usual way with something like this:

<script>
        SystemJS.import('app.ts');
    </script>

SystemJS thinks that this app.ts is DIFFERENT then the app.ts in the bundle because JSPM foolishly assumes everything is it's sibling, and constructs the bundle paths accordingly. And because those paths in jspm.browser.js are used as identifiers for the modules, SystemJS doesn't know their supposed to be replacements for your source files, not additions.

Similar issues have come up. I've been using the browser-sync routes workaround, but obviously that has limitations.

zakdances commented Apr 11, 2016

Yeah, I'll see if I can make a test case for this. Then I'll make a separate issue?

I think this problem is caused by having your source files in a different directory then jspm.browser.js.

My theory: When you bundle, inject, watch, then launch your app in the usual way with something like this:

<script>
        SystemJS.import('app.ts');
    </script>

SystemJS thinks that this app.ts is DIFFERENT then the app.ts in the bundle because JSPM foolishly assumes everything is it's sibling, and constructs the bundle paths accordingly. And because those paths in jspm.browser.js are used as identifiers for the modules, SystemJS doesn't know their supposed to be replacements for your source files, not additions.

Similar issues have come up. I've been using the browser-sync routes workaround, but obviously that has limitations.

@zakdances

This comment has been minimized.

Show comment
Hide comment
@zakdances

zakdances Apr 11, 2016

I made a test case that demonstrates this issue. Let me know if it works.

This issue might also be the same one referenced here (with more tests available here from @kdex).

zakdances commented Apr 11, 2016

I made a test case that demonstrates this issue. Let me know if it works.

This issue might also be the same one referenced here (with more tests available here from @kdex).

@abhipanda

This comment has been minimized.

Show comment
Hide comment
@abhipanda

abhipanda May 25, 2016

Hello Guys,

I tried the build process and everything is working fine for JS files but for html files the update is not working.

Raised an issue for this:
Bundle build not refreshing / updating html files when changed
#1848

Does anyone know how to solve it ?

abhipanda commented May 25, 2016

Hello Guys,

I tried the build process and everything is working fine for JS files but for html files the update is not working.

Raised an issue for this:
Bundle build not refreshing / updating html files when changed
#1848

Does anyone know how to solve it ?

@jonricaurte

This comment has been minimized.

Show comment
Hide comment
@jonricaurte

jonricaurte May 25, 2016

@abhipanda @guybedford I think this issue should be closed since there is Watchman integration now which makes reloading bundles almost instant compared to what was used previously for watching bundles.

jonricaurte commented May 25, 2016

@abhipanda @guybedford I think this issue should be closed since there is Watchman integration now which makes reloading bundles almost instant compared to what was used previously for watching bundles.

@abhipanda

This comment has been minimized.

Show comment
Hide comment
@abhipanda

abhipanda May 25, 2016

It should be.

For the following issue though, we need to add an * at end of the interpreted URL
#1848

builder.invalidate(toFileURL(file));

to

builder.invalidate(toFileURL(file)+'*');

abhipanda commented May 25, 2016

It should be.

For the following issue though, we need to add an * at end of the interpreted URL
#1848

builder.invalidate(toFileURL(file));

to

builder.invalidate(toFileURL(file)+'*');

@born2net

This comment has been minimized.

Show comment
Hide comment
@born2net

born2net Jun 1, 2016

for those of you looking for faster dev times, I recommend hot-loader.
added it to angular2:
Angular 2 Kitchen sink: http://ng2.javascriptninja.io
and source@ https://github.com/born2net/ng2Boilerplate
Regards,

Sean

born2net commented Jun 1, 2016

for those of you looking for faster dev times, I recommend hot-loader.
added it to angular2:
Angular 2 Kitchen sink: http://ng2.javascriptninja.io
and source@ https://github.com/born2net/ng2Boilerplate
Regards,

Sean

@abhipanda

This comment has been minimized.

Show comment
Hide comment
@abhipanda

abhipanda Jun 21, 2016

I have been using the new builder in JSPM 17 beta - it works perfectly and I have some 300 files.

abhipanda commented Jun 21, 2016

I have been using the new builder in JSPM 17 beta - it works perfectly and I have some 300 files.

@tycho01 tycho01 referenced this issue Jun 25, 2016

Closed

jspm/systemjs #3

@Munter

This comment has been minimized.

Show comment
Hide comment
@Munter

Munter Jul 11, 2016

Hi everyone. I finally got around to finishing the tool I had in mind building for a year or so. I would be really happy if anyone in here, who obviously have dealt with load time performance would give it a go and tell me if it helped them out.

It's an express server middleware that intercepts requests to modules requested by systemjs, then bundles them up with all their dependencies on the fly. In the project we implemented it in we saw massive load time speed ups on initial load and especially in incremental updates. I hope you will see the same performance increase and report any bugs you might run into.

https://github.com/Munter/express-systemjs-translate

Munter commented Jul 11, 2016

Hi everyone. I finally got around to finishing the tool I had in mind building for a year or so. I would be really happy if anyone in here, who obviously have dealt with load time performance would give it a go and tell me if it helped them out.

It's an express server middleware that intercepts requests to modules requested by systemjs, then bundles them up with all their dependencies on the fly. In the project we implemented it in we saw massive load time speed ups on initial load and especially in incremental updates. I hope you will see the same performance increase and report any bugs you might run into.

https://github.com/Munter/express-systemjs-translate

@ArmorDarks

This comment has been minimized.

Show comment
Hide comment
@ArmorDarks

ArmorDarks Aug 31, 2016

I've tested 0.17@beta with jspm build --watch flag it seems to be working insanely fast, that's really great. However, I didn't find easy way to hook it up with Browser Sync without launching two parallel processes in console. I guess, it won't be possible, unless jspm's build --watch gets API support? Or am I missing something?

ArmorDarks commented Aug 31, 2016

I've tested 0.17@beta with jspm build --watch flag it seems to be working insanely fast, that's really great. However, I didn't find easy way to hook it up with Browser Sync without launching two parallel processes in console. I guess, it won't be possible, unless jspm's build --watch gets API support? Or am I missing something?

@peteruithoven

This comment has been minimized.

Show comment
Hide comment
@peteruithoven

peteruithoven Aug 31, 2016

Contributor

@ArmorDarks You can easilly combine them in one npm start script, for example:

"start": "jspm bundle src/js/index.js -wid & serve & chokidar-socket-emitter",
Contributor

peteruithoven commented Aug 31, 2016

@ArmorDarks You can easilly combine them in one npm start script, for example:

"start": "jspm bundle src/js/index.js -wid & serve & chokidar-socket-emitter",
@ArmorDarks

This comment has been minimized.

Show comment
Hide comment
@ArmorDarks

ArmorDarks Aug 31, 2016

@peteruithoven Thanks for a hint. I've already tried it. Unfortunately, & doesn't work on Win machines. node-run-all modules, though, solved it and at least it works. Thanks.

I'm just wondering are there any plans to do this in more sane way? While this works, it still looks like a hacking around, especially if you're using Grunt or Gulp. It looks really strange, when part of task runs through Grunt or Gulp, and other part should be run via npm, or hacked around with shell tasks for Grunt or Gulp.

ArmorDarks commented Aug 31, 2016

@peteruithoven Thanks for a hint. I've already tried it. Unfortunately, & doesn't work on Win machines. node-run-all modules, though, solved it and at least it works. Thanks.

I'm just wondering are there any plans to do this in more sane way? While this works, it still looks like a hacking around, especially if you're using Grunt or Gulp. It looks really strange, when part of task runs through Grunt or Gulp, and other part should be run via npm, or hacked around with shell tasks for Grunt or Gulp.

@oligot

This comment has been minimized.

Show comment
Hide comment
@oligot

oligot Aug 31, 2016

Browser sync has some docs on how to use it with Gulp and Grunt.
You could then combine these with the Jspm API to achieve the same result, but without calling the CLI directly.

The CLI (for both Jspm and Browser sync) seems more appropriate when using npm scripts exclusively, while the Node.js API seems more suited for Gulp/Grunt.

oligot commented Aug 31, 2016

Browser sync has some docs on how to use it with Gulp and Grunt.
You could then combine these with the Jspm API to achieve the same result, but without calling the CLI directly.

The CLI (for both Jspm and Browser sync) seems more appropriate when using npm scripts exclusively, while the Node.js API seems more suited for Gulp/Grunt.

@oligot

This comment has been minimized.

Show comment
Hide comment
@oligot

oligot Aug 31, 2016

Oops, I didn't see that the watch mode is not available in the Jspm API.
There seems to be another way to do it as described in this issue: #1702

oligot commented Aug 31, 2016

Oops, I didn't see that the watch mode is not available in the Jspm API.
There seems to be another way to do it as described in this issue: #1702

@ArmorDarks

This comment has been minimized.

Show comment
Hide comment
@ArmorDarks

ArmorDarks Sep 1, 2016

@oligot Thanks. Yeah, no API support for watch so far :(

ArmorDarks commented Sep 1, 2016

@oligot Thanks. Yeah, no API support for watch so far :(

@pcompassion

This comment has been minimized.

Show comment
Hide comment
@pcompassion

pcompassion Sep 2, 2016

What if I want to watch multiple app.js at the same time?

Should I run multiple of them?

jspm bundle app1.js -wid
jspm bundle app2.js -wid

pcompassion commented Sep 2, 2016

What if I want to watch multiple app.js at the same time?

Should I run multiple of them?

jspm bundle app1.js -wid
jspm bundle app2.js -wid
@Rush

This comment has been minimized.

Show comment
Hide comment
@Rush

Rush Dec 16, 2016

Trying different approaches I found best performance with:

  • Having express middleware that compiles all javascripts files to ES5/CJS with Babel on the fly and caches the results. I'm using babel options, and source modified time to generate an ETag along with Cache-Control set to always with 0 max-age.
  • Do not use depCache - depCache will add around 400 requests related to loading the babel runtime.

With all requests delivered with 304 status code the load time with 640 files is around ~3s. Good enough for development. and much faster than watchable bundles. If you change a file it will be lazily recompiled.

If anybody is interested in my middleware code I'd be happy to publish it.

Rush commented Dec 16, 2016

Trying different approaches I found best performance with:

  • Having express middleware that compiles all javascripts files to ES5/CJS with Babel on the fly and caches the results. I'm using babel options, and source modified time to generate an ETag along with Cache-Control set to always with 0 max-age.
  • Do not use depCache - depCache will add around 400 requests related to loading the babel runtime.

With all requests delivered with 304 status code the load time with 640 files is around ~3s. Good enough for development. and much faster than watchable bundles. If you change a file it will be lazily recompiled.

If anybody is interested in my middleware code I'd be happy to publish it.

@peteruithoven

This comment has been minimized.

Show comment
Hide comment
@peteruithoven

peteruithoven Dec 16, 2016

Contributor

@Rush I'm definitely interested. So your middleware also watches files? But it only recompiles the modules that changed?

Did anyone get further with http2/spdy push? Expush lacks documentation and hasn't changed since oct 2015 and it's still an open issue on jspm-dev-server.

Contributor

peteruithoven commented Dec 16, 2016

@Rush I'm definitely interested. So your middleware also watches files? But it only recompiles the modules that changed?

Did anyone get further with http2/spdy push? Expush lacks documentation and hasn't changed since oct 2015 and it's still an open issue on jspm-dev-server.

@Rush

This comment has been minimized.

Show comment
Hide comment
@Rush

Rush Dec 16, 2016

@peteruithoven No need to watch files. There's just one fs.stat per request. if the file is modified (its modify time is newer than the cached version) it will be processed on the fly and cached again, otherwise an already cached version will be served.

Rush commented Dec 16, 2016

@peteruithoven No need to watch files. There's just one fs.stat per request. if the file is modified (its modify time is newer than the cached version) it will be processed on the fly and cached again, otherwise an already cached version will be served.

@Rush

This comment has been minimized.

Show comment
Hide comment
@Rush

Rush Dec 16, 2016

@peteruithoven I will package my code into a re-usable module tomorrow. One problem I have is inability to easily to synchronize node's babel options with jspm's .. but I do have some ideas on how to improve it.

Rush commented Dec 16, 2016

@peteruithoven I will package my code into a re-usable module tomorrow. One problem I have is inability to easily to synchronize node's babel options with jspm's .. but I do have some ideas on how to improve it.

@Rush

This comment has been minimized.

Show comment
Hide comment
@Rush

Rush Dec 16, 2016

@peteruithoven HTTP2/PUSH is a no-go for development as Chrome and other browsers will refuse to connect to a non-TLS HTTP2 server (as far as I know).

Rush commented Dec 16, 2016

@peteruithoven HTTP2/PUSH is a no-go for development as Chrome and other browsers will refuse to connect to a non-TLS HTTP2 server (as far as I know).

@peteruithoven

This comment has been minimized.

Show comment
Hide comment
@peteruithoven

peteruithoven Dec 16, 2016

Contributor

@peteruithoven No need to watch files. There's just one fs.stat per request. if the file is modified (its modify time is newer than the cached version) it will be processed on the fly and cached again, otherwise an already cached version will be served.

That sounds like something that might even save system resources?

@peteruithoven HTTP2/PUSH is a no-go for development as Chrome and other browsers will refuse to connect to a non-TLS HTTP2 server (as far as I know).

I know, but you could use a self signed certificate? (I do understand that this is harder to setup)

Contributor

peteruithoven commented Dec 16, 2016

@peteruithoven No need to watch files. There's just one fs.stat per request. if the file is modified (its modify time is newer than the cached version) it will be processed on the fly and cached again, otherwise an already cached version will be served.

That sounds like something that might even save system resources?

@peteruithoven HTTP2/PUSH is a no-go for development as Chrome and other browsers will refuse to connect to a non-TLS HTTP2 server (as far as I know).

I know, but you could use a self signed certificate? (I do understand that this is harder to setup)

@Rush

This comment has been minimized.

Show comment
Hide comment
@Rush

Rush Dec 21, 2016

@peteruithoven took longer than anticipated but delivered as promised https://github.com/virtkick/node-serve-jspm

Let me know if it works for you.. I was able to use JSPM's babel and read per file configuration. There was one issue that I was able to workaround #2202

It requires jspm@beta of course

Rush commented Dec 21, 2016

@peteruithoven took longer than anticipated but delivered as promised https://github.com/virtkick/node-serve-jspm

Let me know if it works for you.. I was able to use JSPM's babel and read per file configuration. There was one issue that I was able to workaround #2202

It requires jspm@beta of course

@Rush

This comment has been minimized.

Show comment
Hide comment
@Rush

Rush Dec 23, 2016

@peteruithoven I upgraded serve-jspm to exclude hard-coded babel reference. Now it will run whatever transpiler that's configured in your jspm project.

Rush commented Dec 23, 2016

@peteruithoven I upgraded serve-jspm to exclude hard-coded babel reference. Now it will run whatever transpiler that's configured in your jspm project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment