Can we speed up server / "both" imports?

[gadicc]

Since #17 we can handle hot loading (on the client) from files that are shared on the client & server, but the latter is a lot slower, e.g. @sammkj in #17 (comment):

😱 I just tested the newest version. It's working perfectly! However it's not nearly as fast as reloading components in client folder.

This is (probably) because all these shared files need to be processed twice before we can send the update. I think the only way we can solve this is to speed up Meteor (preferred) or bypass Meteor and repeat a lot of the same logic. Will look into this when I have a chance.

[proxiper]

As workaround I tried just comment import of "imports/both/routes.jsx" in "server/main.jsx" (in the development mode), but this didn't increase a reload speed. Is it possible to fast reload code in common "imports" folders, but not required by any server scripts?

[gadicc]

@proxiper, I think the reason this doesn't help is related to meteor/meteor#6576, in that Meteor still processes all those files even if they're not imported. Not 100% sure though.

Nevertheless, @ALL, I think we've gotten around this! And even fast client load times. Try this in your .meteor/packages:

gadicc:ecmascript-hot@=1.3.1-fast.3

There's a bit of extra debugging info in the server console for this experimental release, that will go away with the final. It does tell you how much extra time was saved on each change though. Would love some report backs especially from Mac and Windows users, to confirm that it works on these platforms. You should see stuff like:

[gadicc:hot] Saved an extra 3808 ms

if you don't, it still falls back to the old way. So the new way is only working if you see this message.

[gadicc]

Ssory, I should have mentioned, this release is for Meteor 1.3.1. That's not a "recommended" release by MDG yet, so you need to specify by hand: meteor update --release 1.3.1. Otherwise you'll get dependency errors.

[proxiper]

On the first run with @=1.3.1-fast.3 I got this:

=> Starting gadicc:ecmascript-hot server on port 3002.

[gadicc:hot] /Users/anton/meteor-ecmascript-hot-source-map-problem-demo/.meteor/local/build/programs/server/assets/packages/gadicc_babel-compiler-hot/accelerator.js
   Loading plugin `compile-ecmascript-ho...  -
module.js:340
    throw err;
          ^
Error: Cannot find module '/Users/anton/meteor-ecmascript-hot-source-map-problem-demo/.meteor/local/build/programs/server/assets/packages/gadicc_babel-compiler-hot/accelerator.js'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:945:3
=> Errors prevented startup:                  

   While processing files with gadicc:ecmascript-hot (for target web.browser):
   child_process.js:406:26: channel closed
   at ChildProcess.target.send (child_process.js:406:26)
   at Object.hot.forFork (packages/gadicc_babel-compiler-hot.js:442:10)
   at BabelCompiler.BCp.processFilesForTarget (packages/gadicc_babel-compiler-hot.js:584:7)


=> Your application has errors. Waiting for file change.

But it worked well on the second run.

Wow! Now from client folder it reloads even faster then webpack projects! Excellent work!

From common folders it's still much slower, but acceptable slower now.

Thank you!

[gadicc]

Thanks @proxiper for the detailed feedback. Will figure out why that's happening on the first run. Pity about the common folders speed, I'm going to leave this issue open and see when I have time to delve into performance again... probably we'll have to take a bit more control away from Meteor. Glad at least that it's useable though now.

[reel]

Great package... With fast.3, it works but I have the following:

=> Started proxy.
=> Started MongoDB.
=> Starting gadicc:ecmascript-hot server on port 3002.

[gadicc:hot] C:\Users\Felix\code\projectss\.meteor\local\build\programs\server\assets\packages\gadicc_babel-compiler-hot\accelerator.js
   Building for web.browser                  \
fs.js:1063
    throw errnoException(process._errno, 'watch');
          ^
Error: watch ENOENT
    at errnoException (fs.js:1031:11)
    at FSWatcher.start (fs.js:1063:11)
    at Object.fs.watch (fs.js:1088:11)
    at process.lastCall (C:\Users\Felix\code\projectss\.meteor\local\build\programs\server\assets\packages\gadicc_babel-compiler-hot\accelerator.js:12:10)
    at process.emit (events.js:98:17)
    at handleMessage (child_process.js:322:10)
    at Pipe.channel.onread (child_process.js:349:11)
=> Started your app.

=> App running at: http://localhost:3000/
   Type Control-C twice to stop.

And I don't see the [gadicc:hot] Saved an extra ... ms message.

[gadicc]

Thanks @reel, I think you'll only get that the first load. If you reload Meteor one more time it should all work.

all, I know why this is happening... next release will either solve it or at least forego the error message and give instructions to reload Meteor.

[gadicc]

Btw, @proxiper, I noticed that sometimes the first hot reload is a lot slower than consecutive reloads. Could you just confirm in your current project that shared folders take much longer in the 2nd and later reloads? Thanks.

[reel]

@gadicc Yes it works.

First reload is a bit slower. After, the reload time is +/- 6000ms for subsequent changes...

[gadicc]

Wait, so, 1st reload is extra 6s and 2nd+ is fast?

[gadicc]

New release: gadicc:ecmascript-hot@1.3.1-fast.5 (experimental)

Please report if this works for you and what OS you're on!

Yet another different approach, stuff happens completely outside of Meteor, so should be even faster and unaffected by speed probs in shared folders. On the downside, harder to maintain, and more error prone, so please help by letting me know if it works, how well it works, and any issues.

[gadicc]

Quick note, in some circumstances the "accelerator" which runs on your meteorPort+2, will survive Meteor exiting (generally if Meteor dies from an unusual error). I think I've fixed this now, will see how things go. If this happens to you (on a new Meteor load you get Error: listen EADDRINUSE and/or child_process.js:406:26: channel closed), for now, you'll need to find this run-away process and kill it manually, e.g.

$ ps x | grep accelerator | grep <port_num>
$ kill <pid>

Where port_num is the accelerator port (look out for => Starting gadicc:ecmascript-hot server on port 7002 when you start Meteor), and pid is the process id from the ps command. If you didn't understand any of that, just reboot (sorry).

[gadicc]

My suspected fix is published in gadicc:ecmascript-hot@1.3.1-fast.6 (5bde6f4)

[proxiper]

I tested gadicc:ecmascript-hot@1.3.1-fast.6 with flow-router-ssr and it works great! Thank you!

I have only one small issue:
hot-client.js:180 Uncaught TypeError: Cannot read property 'm' of undefined(anonymous function) @ hot-client.js:180walkFileTree @ hot-client.js:128walkFileTree @ hot-client.js:131meteorInstallHot @ hot-client.js:179(anonymous function) @ hot.js?hash=eedd690…:1

This occurs when the server is restarting due to changes in server methods. But after the restart everything continues to work.

I'll try this version in a bigger real project and will give my feedback again.

[gadicc]

Great news! Thanks for reporting back. What OS are you on, btw?

Let's track the other issue in #35.

[gadicc]

Closing since I achieved all my aims here ^_^