Updating the app source and node-webkit separately

[adam-lynch]

This has come up here, nwutils/nw-builder#71 and elsewhere before (I can't find some issues where it was discussed).

The upsides are that you have smaller releases, quicker updates, etc.

---

From looking at Popcorn Time's source, they do this. Hopefully @sashahilton00 (who worked on it) can clear up how it works and any downsides.

When you run theirinitial installer, you get Popcorn Time.exe (which is actually node-webkit.exe renamed) in a node-webkit directory, a shortcut called Popcorn Time (in the root next to their app package.json manifest) which just calls Popcorn Time.exe. The actual app code is in a src directory.

They seem to check for updates just like we do, but then only download a ZIP containing a .nw. When the user is alerted that the new version is ready and they click to restart, the same original app (Popcorn Time.exe) is re-ran (i.e. spawns process.execPath and quits) but passes the newly extracted .nw file as an argument. So the new app (.nw) runs from the download location with the original .exe.

How does it copy itself over the old original app code in src? I don't see where in the source this is handled. Since the new app is running from the downloaded location, I assume this is an intermediary step like we have where it then copies itself to the original location and restarts itself again but this time just using the src directory.

Is it something to do with the .old and .new suffixes? The new version is extracted as *.nw.new or something like that. When preparing the update for linux the name is renamed as I would expect, but not for Windows. Not sure what's going on here or how the existing app sources are udpated.

What if node-webkit itself needs to be updated? I.e. you have yourApp-0.5 installed (which was installed with node-webkit@0.10.5) and an update is available to yourApp-0.6 which needs node-webkit@0.11.0.

Are there any downsides to this approach?

---

Also to note

• They don't have single-instance set to false (Using "single-instance": false to avoid problems #62)
• @sashahilton00 said in Security of packages #56 that their update system isn't 100% solid.

[sashahilton00]

Hi @adam-lynch,

it looks like they have rewritten the code since I last worked on it, but the core of it is still there. For copying over the old app code, this never actually happens. As we are running the app in node-webkit, the entire application (app.nw) is loaded into memory at boot, so the files in app.nw are not being actively processed. if you then look here: https://git.popcorntime.io/stash/projects/PT/repos/popcorn-app/browse/src/app/lib/updater/index.js#227 the code is actually pretty simple, all we do is delete the app.nw that needs to be updated, and replace it with the new app.nw that we have downloaded. (Note, the app is still running the old app.nw), hence the reboot. Once rebooted, the node-webkit binary looks for app.nw to load into memory, at which point it loads our newer version of the app, and the update is complete.

I don't work with linux, but I imagine that they just haven't updated the code for linux yet, or it is something to do with permissions, or the way the binary for linux behaves, I'm afraid I don't really know. For Windows and Mac, it uses the process as described above.

The updating of node-webkit is not covered at all with this updater. At some point, I have been meaning to get a working version of the node-webkit source working on my mac, add the Sparkle framework to the project, then let it crunch out a binary for a day or so ;) After that, I seem to remember that Sparkle could be modified to do silent updates, and could accept partial-updates (think Google Omaha), but I would have to confirm this.

Depends which approach. The inbuilt updater is much easier to implement, but seemed to be a bit fragile, though I don't know if it still is after the code rewrite. The Sparkle solution has the advantage of thousands of hours of devs putting work in for it, and allows us to update the entire application, and is significantly more robust, at the expense of complexity. Personally, I would be pushing for PT to start using Sparkle, as in the long run, it will be much more robust than the current solution, and I would hope that they implement it before it comes out of beta, but for the current stage of development, it makes sense to keep it simple until we have a more stable desktop app that everyone is settled on.

[adam-lynch]

For copying over the old app code, this never actually happens.

So the new .nw is downloaded to temp or somewhere and the .exe is spawned using this, but what then? Like if you exit the app and use the shortcut, won't it use the src directory? (Not the newest code in temp).

. As we are running the app in node-webkit, the entire application (app.nw) is loaded into memory at boot, so the files in app.nw are not being actively processed.

Ah, wait, so you're saying it is copied over the old app source. The contents of src is deleted and the new .nw is extracted into it. But look at installWindows, it doesn't seem to do the same thing. In fact, it seems to extract the ZIP where it was downloaded (in temp or whatever) so this is the path that will be passed to the .exe when it restarts. This is why I was confused in my first paragraph in this comment.

The updating of node-webkit is not covered at all with this updater.

:/ Ok. I'll look into this type of update system with the addition of updating node-webkit separately.

At some point, I have been meaning to get a working version of the node-webkit source working on my mac, add the Sparkle framework to the project, then let it crunch out a binary for a day or so ;) After that, I seem to remember that Sparkle could be modified to do silent updates, and could accept partial-updates (think Google Omaha), but I would have to confirm this.

I've never heard of Sparkle or Google Omaha 😄. Correct me if I'm wrong but Sparkle looks like it's for Mac only and we thought about having "One desktop autoupdate client shared by all desktop client software" like Omaha describes itself using node-webkit (i.e. having a second app) instead but we decided not to because this updater exectuable would be too big.

But this may need to be rivisited for updating the node-webkit executable itself (since if only a .nw is distributed then a lot of weight is gone).

The inbuilt updater is much easier to implement, but seemed to be a bit fragile.

For me, we're only using node-webkit-updater currently on Windows but it is solid.

[sashahilton00]

The windows version doesn't run the app from app.nw, rather it runs it from the 'app' folder. The temp download location for windows is the same directory as the 'app' folder. The app.nw is then unzipped into the 'app' folder, overwriting the contents of the previous app, then the downloaded file is deleted.

Yes, sorry I forgot to reference that; the sparkle update framework would only be for the Mac version, whilst a different update framework would be needed for Windows/Linux. This is why I hinted at Omaha. I am still looking into cross-platform update frameworks, but so far not much luck.

I was talking about the PT version of the updater. This version does seem to be more stable, but I have occasionally had issues with the test app that comes with this repo, though I do modify the code before building, so don't take it too seriously.

[adam-lynch]

The windows version doesn't run the app from app.nw, rather it runs it from the 'app' folder.

So you mean src/app?

I was talking about the PT version of the updater.

Yeah I got that, just saying I think it's possible to have a solid built-in (node-based) updater as long as it has all the right checks and stuff.

but I have occasionally had issues with the test app that comes with this repo, though I do modify the code before building, so don't take it too seriously.

There could be something small wrong with it. I haven't actually used it. If you come across anything wrong with it again, let us know.

---

@edjafarov do you think this is something that belongs in this project or should it be its own one? So far, this project is intended for apps which are built with node-webkit-builder or similar (where you have one .exe initially and per update), but this would be drastically different.

(That's if it's even possible to have one third-party project which will handle separate app and node-webkit updates separately).

[sashahilton00]

No, when I say the 'app' folder, I mean the one that the installer creates, located somewhere like: /Appdata/.../Your-App-Name/app/, which houses all of the files for the node-webkit app.

I agree, I think it is easily possible to have a robust updater in nodejs, but the current one (in PT) needs some work. node-webkit-updater seems to be a bit more robust, but then again, the apps I have used it for don't have the same user base as PT...

Regarding solutions for this in the future, what I suggest is a v. small 2nd application that runs as a service at startup. That way, a nw app could pass the parameters in a json file of what needs to be updated (location, downloadurl, version, etc.), and the service could download all of the stuff and replace the files in a much more reliable manner, as opposed to the node hackery that is being used atm to install in app upgrades. The second advantage of having a second app as a service is that the app could also pass a parameter to it that requires an upgrade to the actual nw binary, thus we solve the problem of in-app vs. full app upgrades... give me your thoughts ;)

[edjafarov]

@adam-lynch that was original Idea to have such flexibility and update sources in a runtime without reload of app. But I ended up with what we have right now because it is 100% bulletproof.

On other hand can we diff the resulting binary and send the binary diff to the updater so that updater applied it and we would have same lightweight releases in more uniform way across Os'es with less efforts. What do you think? It could make sense to create POC.

[edjafarov]

Something like http://www.daemonology.net/bsdiff/ for Win, add em as we do for unzip. Same for unix. For mac it is easier since app for OSx is just a folder, we can replace js/css source.

thoughts?

[adam-lynch]

@sashahilton00

No, when I say the 'app' folder, I mean the one that the installer creates, located somewhere like: /Appdata/.../Your-App-Name/app/, which houses all of the files for the node-webkit app.

This is what belongs in .../AppData/Local/Popcorn Time:

So, can you clarify exactly what you mean? This folder, the inner src directory or src/app.

---

@edjafarov

But I ended up with what we have right now because it is 100% bulletproof.

Why isn't the other method bulletproof though?

About diffing the executable, that sounds cool but I don't know much about native modules. I've two reservations:

• I don't like using modules that depend on node-gyp
• I'd like the updater to be written in the language the users of the updater use. I think that's the biggest problem is node-webkit, I'd try contribute a lot if I knew C++ but of course, I use JS / Node.

I'd love a solution that is done with node / node-webkit.

What I really like about node-webkit-updater right now is that everything is contained in your own app and interlinked with your own code. Plus how the one app is then used in multiple ways in the update step so we only need one item per update (the new executable).

Ideally, I'd like a mix between what node-webkit-updater does now and what Popcorn Time does. So, updating the app code by distributing updates as .nw and having a separate way of updating node-webkit itself if the new .nw specifies it needs a different version of node-webkit. Hopefully while figuring out a way of avoiding a second update client for updating node-webkit.

---

@sashahilton00, @edjafarov

I just thought of something. A folder / .nw file can be ran with any node-webkit app, not just the original node-webkit.exe. So what if the .exe just contained simple code for updating itself.

Typically, just like Popcorn Time, a .nw / folder is passed to the .exe. If the app code needs to be updated, it uses the Popcorn Time method. If node-webkit needs to be updated (the new update manifest should indicate this) then the .exe is restarted without a folder / .nw being passed and it'll use the node-webkit-updater method of updating itself.

The only potential problem I see is when updating node-webkit; do you update the app code or node-webkit first? If you update the app code first, it might fail because it's trying to use a non-existent node-webkit function for example. If you update node-webkit first (which is probably the way to go), then there is still a chance that something in the code might not work with that version. Example: A regression in node-webkit could've been added; for example in the newer version, calling notification.close() crashes the app so in your new update, it doesn't call it. I think this might be a separate issue though... see below.

It should also be kept in mind that it is possible that an update of your app might depend on an older version of node-webkit than the previous version.

---

Rolling back

No matter what you go with, the app might crash. It doesn't matter how solid the updater is if your new app crashes on launch or something like that because of a missing DLL / incompatibility with that version of node-webkit on that platform, etc. I think a useful feature of node-webkit-updater might be a rollback feature.

I.e. Before an update is done, the entire app (node-webkit and app code) is backed up to another directory.

So there may need to be a check after the new app is ran and on failure, the app is rolled back and a report is sent. Not sure exactly how this should work; if a file should be stored somewhere which is removed when the new app has worked and something else checks if that file still exists, etc.

Or maybe when the app code is updated, then it actually stores the code in a different directory than it normally does, re-writes the shortcut to point to there and after the next time the app is successfully launched, it copies the code over to the right directory and re-writes the shortcut back. The same could be done with the node-webkit version.

[sashahilton00]

Just a few thoughts about this:

I'm not sure a rollback feature is necessary or particularly helpful to the developer. A developer's worst nightmare is having a fragmented user base, and allowing people to roll back is not necessarily a good idea.

Regarding the updater, I have been looking into possible solutions, and I think there may be some potential by using something like forever or daemonize2, and launching a separate instance of a node updater that runs as a service once the app has started. Here is the workflow:

Node app starts -> node app checks for forever service, and if not running, starts the forever service -> Node app checks for updates -> If there are any updates, node app passes the relevant details to the forever service -> forever service downloads the required stuff, then sends an ok response the the node app -> node app alerts user that there is an update -> user presses update, at which point the app notifies the forever service to proceed with the update, then closes -> forever service deletes old files (be it the .nw file, or the node-webkit binary), and replaces it with the new ones -> forever service relaunches app, which has now been updated.

If we decide to go down the route as described above, the entire thing can be done in nodejs, and we will be able to update both the binary and/or the .nw file independently, and consistently.

@adam-lynch I don't have windows, but I think I mean the src/app directory.

[adam-lynch]

I'm not sure a rollback feature is necessary or particularly helpful to the developer. A developer's worst nightmare is having a fragmented user base, and allowing people to roll back is not necessarily a good idea.

To clarify, this is for the developer, not the user. The developer defines what is deemed a "working" app. If that is not met after a new update, it's not applied, they get alerted of the failure (with the cause), they make a patch and issue a newer update. There would be nothing worse than issuing an update that crashes a user's app and they can't update to the next (fixed) version because their app crashes before it gets the chance to.

Again though, this is a whole separate issue.

---

Wouldn't there be lots of issues with permissions and stuff with services, etc? Is it possible that some user's companies could block apps from creating services or anything like that? The implementation might be a good it different per platform; forever doesn't work on windows, instead you might use something like node-windows.

Node app starts ->

I assume here you're talking about a node-webkit app. So the service can alert the nw app when there's an update? That's important so that information about updates can be shown inside the node-webkit app (like I have stuff on my about screen and other places right now).

---

I'm not sure why but I'm a bit sheepish about something that runs in the background. I like how right now everything is contained within the one app.

Even if the updating logic is broke in a new version of the app (where two thirds of logic for the updating process is contained)... let's say the copying of the new app over the old is broke. Then you can just issue another update with the update logic fixed. With something running in the background, I'd be worried what if it would break, if the user could get stuck in limbo and not be able to update, etc.

---

@edjafarov It seems either way that we're thinking of using a method which separates the updating of the executable and the app code. I'm not sure it belongs in this project though. If we were to change this project then lots of users would have to change how they build their app (no longer with node-webkit-builder, etc.) and if there's config in this module to switch between one or the other then there would be needless code and dependencies in user's apps. I'm leaning towards the idea of creating another project with this type of update system, maybe with a common module used by both which contains utility functions, I don't know.

Maybe there could be a common spec / API between this updater, @sashahilton00's idea and my idea. Just thinking out loud here.

[edjafarov]

@adam-lynch you might be right. At least node-webkit-updater works. The other project could start as an experiment and if it will prove the idea and will be stable enough one will be able to choose.

[sashahilton00]

@adam-lynch @edjafarov Ok, this conversation has been a useful source of ideas for a potential updater that will be more flexible, and hopefully robust. I will try to make time to create a repo and wok up a simple testing app to see if this concept works... though it may not happen immediately ;)

[sashahilton00]

@adam-lynch I'm not sure if there will be permissions errors or not, my hope is that because the rocess is spawned by the node app, the permissions should be fine... regarding the stuck in limbo problem, I actually had that in PT, even though the updater is self contained, it deleted the old files, then failed to copy across the new files due to a spelling error... no solution is going to prevent dodgy updates from screwing up an application, the best we could do would be to copy the old files to a temp dir in the updater process before overwriting with the new ones, at least that way, we could roll it back if the app has a bad update...

[ghost]

I actually rewrote the updater (the original version was written by @jduncanator so he might be able to add some more insight into this) so if you have any questions, feel free to ask me!