Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ugly debug upgrade tool #1260

Closed
ErisDS opened this issue Oct 24, 2013 · 31 comments

Comments

Projects
None yet
6 participants
@ErisDS
Copy link
Member

commented Oct 24, 2013

I would like to start investigating adding an upgrade tool for Ghost. There's no UI yet so it can live on the ugly debug tools page.

Part 1: would be to have a 'check for updates' button. It should be possible to make a request to http://ghost.org/zip/ghost-latest.zip, then read and parse the location header on the response which will be something like http://d36u7lo2kegj1p.cloudfront.net/archives/ghost-0.3.3.zip

We can then match the version number from the location header, with the version number in package.json to determine whether or not there is an upgrade.

Part 2: would be to have a second 'Upgrade Ghost' button appear if an update is found.

A backup of the database should be made using exporter before proceeding. The upgrade button should then make a http request to fetch the http://ghost.org/zip/ghost-latest.zip zip file into a freshly created /content/tmp/ folder and unpack the archive. The Ghost core/ folder and root files should then be copied over the top of the existing install.

The user should be presented with an option (probably a checkbox above the 'Upgrade Ghost' button which is selected by default) to choose whether or not the Casper files are also updated.

Finally, once the files have been successfully copied into place, a message should be displayed clearly indicating that the user must manually restart Ghost before the changes will take effect.

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Oct 29, 2013

Ok so I got to start looking into this a bit more heavily tonight. Here's a brain dump of what I'm thinking for general architecture.

For now:

  • Add new route GET /api/v0.1/update/ which will return an object of the below structure. The request will perform the operations suggested in Part 1 above.

    {
     "isUpdateAvailable": true,
     "currentVersion": "0.3.3",
     "newVersion": "0.4.0"
    }
  • Add new route POST /api/v0.1/update/ which will return an object of the below structure. The request will perform the operations suggested in Part 2 above.

    {
     "success": false,
     "detail": "Write access to disk was denied."
    }
  • The UI on the debug page will be a button that invokes an GET request (not sure if a backbone model is really desired for this or not) to /api/v0.1/update/ to determine if an update is available. If an update is available, the user will be presented with a confirmation dialog. Should they click "yes", we will close the dialog, send a POST to /api/v0.1/update/, start the busy indicator, and display a relevant notification at the end of the process.

For the future:

  • Logging into the admin interface should check for an update.
  • When performing an update, open a websocket for near-real time progress of the upgrade.
@sebgie

This comment has been minimized.

Copy link
Contributor

commented Oct 29, 2013

👍 for the API calls

Two things that came to my mind:

  • routes are now /ghost/api/v0.1/...
  • How can we determine that the update process finished and Ghost is up and running again?
@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Oct 29, 2013

The user has to be told to restart Ghost at the end of the process, we can't do it for them cos we don't know how they are running it (forever, service, etc)

@hswolff

This comment has been minimized.

Copy link
Member

commented Oct 29, 2013

Would it make sense to incorporate moving Ghost to be used as an npm module as part of the upgrade process?

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Oct 29, 2013

Ghost is an app, not a module, using it as a module is a non-standard use case, so upgrading that way would only serve a small fraction of users?

The Ghost upgrade process being implemented here is for all of our users who use Ghost as an app.

@hswolff

This comment has been minimized.

Copy link
Member

commented Oct 29, 2013

I remember reading an issue that said an eventual goal is to move Ghost to be used as an npm module, npm modules having its own upgrade process.

Only reason I would even think this is appropriate for the average user is because you mentioned requiring user action to restart the Ghost process.

If an npm module process is used you could have the upgrade script rewrite the package.json file and run npm install, however I'm not sure how 'kosher' of a process that is.

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Oct 29, 2013

I understand how npm works, and putting Ghost into npm is something we're working on, but that's still an entirely different use case to the one being managed by the upgrade process we are implementing in this issue.

The user action to restart is a separate issue which we can and will mitigate in various ways, but would not in anyway be resolved by upgrading through npm and is an enhancement to the issue being described here.

@hswolff

This comment has been minimized.

Copy link
Member

commented Oct 29, 2013

Fair enough! :)

This issue is mostly for end-user flow, as opposed to dev flow perhaps?

Is the npm module going to be solely for dev flow or will it also eventually replace this update flow? Forgive my misunderstanding, seems like this issue is perhaps a (good) stop-gap along the way towards npm modulization? Cuz moving Ghost to work as an npm module ain't going to be no picnic.

(also apologies for getting off topic 🐼 )

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Oct 29, 2013

They are two different use cases for different kinds of users, they are both valid use cases, and both can live side by side.

Perhaps you can expand on your thoughts around the difficulties of making Ghost function as a module in #1326 - I assume this is what you are referring to, not the act of publishing it on npm?

@hswolff

This comment has been minimized.

Copy link
Member

commented Oct 29, 2013

Assumption correct, I'll kick around some thoughts in that issue.

Also what different kinds of users did you have in mind for these two use cases?

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Oct 29, 2013

There are many ways people can and will get started with Ghost, some download from Ghost.org, some clone the repo, yet more still use the various one-click style installers on various hosting platforms. Every single one of those users needs to be able to upgrade Ghost - that is what this upgrade is for - if you're using Ghost right now, upgrading takes just a couple of commands, but it's still a complete PITA and very error prone.

Using npm to install and upgrade Ghost is going to be useful for devs, especially those who want to use Ghost as part of a larger application and also for the kind of non-dev user that is happy following a command-line tutorial to get up and running.

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Oct 30, 2013

Quick update: I made some significant progress on this tonight and it should be ready for a look-see here shortly (I hope). Just need to wrap up the unzip/cleanup process -- I ran into an issue with zlib/node-unzip where I'm getting "Invalid Block Type" (error code is "Z_DATA_ERROR") when extracting the contents, but windows unzips the same file fine.

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Oct 30, 2013

@gotdibbs I've run into that issue before - we had to switch what we used for doing the builds because of zips that would only unpack on Windows. grunt-contrib-compress which we use for builds now uses https://github.com/ctalkington/node-archiver and that seems to work.

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Oct 30, 2013

@ErisDS Sadly node-archiver only seems to compress as far as I can tell. It also uses zlib under the hood which is what node-unzip is using as well :(

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Oct 30, 2013

Figured out zips by using adm-zip and things look to be moving now.

My one last hangup is whether or not we need to run npm install and npm update. I'm assuming we should, but from the trials I've had with using child_process to handle this, it doesn't seem to handle errors well. So I could run it but I wouldn't have any idea, if it failed, why it failed.

Update:
So I found another way to execute npm commands from node. Apparently you can actually require npm? Who knew.

However, I don't think we can actually do this currently. We get an error on trying to update sqlite3 while Ghost is running, presumably because of a file lock.

I'm not sure how we want to message this in the user interface, but I will submit a PR with what I've got and I'd appreciate peoples looking it over and providing any feedback they can. I've also attached a gif of what the UI flow looks like as of now below.

update

gotdibbs added a commit to gotdibbs/Ghost that referenced this issue Oct 31, 2013

Initial auto update implementation
Fixes TryGhost#1260

- Added update button to ugly debug UI.
- Button check for updates by parsing the location header on the
ghost-latest.zip redirect.
- If an update is available, a modal confirmation window is presented to
confirm the installation of the update.
- Installation process includes creating the `content/tmp/` folder
(unless it already exists), downloading the zip to that folder,
extracting the zip on top of the current installation and then deleting
the zip from `tmp`.
- Note the update process still requires you to manually run `npm
install` and `npm update` and restart the server. See the comments in
the associated item for more information to that end.

gotdibbs added a commit to gotdibbs/Ghost that referenced this issue Nov 9, 2013

Initial auto update implementation
Fixes TryGhost#1260

- Added update button to ugly debug UI.
- Button check for updates by parsing the location header on the
ghost-latest.zip redirect.
- If an update is available, a modal confirmation window is presented to
confirm the installation of the update.
- Installation process includes creating the `content/tmp/` folder
(unless it already exists), downloading the zip to that folder,
extracting the zip on top of the current installation and then deleting
the zip from `tmp`.
- Note the update process still requires you to manually run `npm
install` and `npm update` and restart the server. See the comments in
the associated item for more information to that end.
@hswolff

This comment has been minimized.

Copy link
Member

commented Jan 16, 2014

So some updates from IRC land cross posted for even greater prosperity. Plucked the salient bits: (direct link to conversation)

<HannahWolfe> and consider things like
<HannahWolfe> verification via hash (we can return that from updates.ghost.org)
<HannahWolfe> whether it should be done in a different proces
<HannahWolfe> and how, if at all, we could restart Ghost to complete the process - or complete the process without a restart
<HannahWolfe> the issue about having write access making stuff less secure is worth noting as well
<HannahWolfe> it would be good to check *if* ghost has write access to itself and disable the button if not - and perhaps also keep the funciton abstracted out so we can make a cli tool to do it instead
<HannahWolfe> would like to have commander or something cli tools for migration, upgrades, and other useful stuff
<HannahWolfe> So I think, first things first, we need to make Ghost as savvy as possible about whether it can and should do an upgrade cos yes, how we check for upgrades has changed, and also we want to check permissions.
<HannahWolfe> Then there's the question of the best way to do the upgrade
<HannahWolfe> I had wondered if we shouldn't do an actual diff on files, remove any that are no longer present, add any new ones, and replace ones whose hashes have changed

and

<hswolff> may need a managing process
<hswolff> that'll handle if ghost needs to be restarted
<hswolff> well... if you have Ghost/index.js spawn the Ghost/core/index.js process
<hswolff> and then that's self contained in node land

Also had the thought that perhaps some node version of rsync may be a good fit for updating the files perhaps?

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Jan 17, 2014

Thanks @hswolff for copying over the summary of what we discussed.

One thing I was thinking about today: do we really need to automatically restart Ghost ourselves? I would think process management should not be handled by us. My assumption is that process management is handled by other tools like forever and IISNode and the like anyways.

If we can make that assumption/recommendation safely, then we get out of the need to restart Ghost ourselves, we would just kill the process and let the process manager handle the restart.

Of course the goal should still be to not need a restart but I think the topic of process management deserved calling out for discussion anyways.

What are the community's thoughts?

@hswolff

This comment has been minimized.

Copy link
Member

commented Jan 18, 2014

I don't think we can safely assume someone will use a process management tool when instantiating Ghost.

For example my current production set-up has a cron task that runs every 20 minutes and checks if the node process is running, otherwise it executes it again.

As @ErisDS pointed out there's two main aspects of updating Ghost:

  1. Fetching the new version code and writing it to the local file system.
  2. Having Ghost use that newly updated code.

On point 2 I can envision two main ways of achieving this.

  1. Manage the main Ghost process and allow for a way to signal that it should restart the main Ghost process.
  2. Re-digest all require'd files so that those are then used.

I have no idea if the latter is possible TBH.

/braindump

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Jan 21, 2014

My personal opinion is it's bad to not promote that some sort of process manager be in charge of the Ghost process.

Either way, I've given this a lot of thought and have come to some conclusions:

  1. It does not seem feasible to be our own process manager, in charge of spinning up Ghost as a daemon/service -- at least not with as many platforms as we support.
  2. Re-digesting our own files might be possible, but I can guarantee we'll be dependent on npm packages which will change and there's no great way that I can see to invalidate the cache and that, it appears, is intentional.
  3. I sincerely believe command line tools for doing the update would be going down the wrong road. The less user-friendly updates are, the less likely the update is to take place which of course is at the very lease a security concern. We all know that command line tools are not internationally recognized as user-friendly :).

Therefore something similar to what is already in my first pass may be our best bet. We need to warn the user that starting an update will require them to restart the Ghost process after the update has been completed.

I might even go so far as to suggest, process manager or not, that we kill the blog the moment the update completes so we don't have anything running in a weird intermediary state.

If anyone has any great ideas that would help with this, I'd love to hear them, but I don't see too many options for us given node's current architecture and the breadth of our cross-platform support.

@hswolff

This comment has been minimized.

Copy link
Member

commented Jan 21, 2014

@gotdibbs my original thought was not to create a full fledged process manager, rather just a super process that creates a child process fork and that is what actually starts the Ghost blog.

If I have time I may try that out to see if it is truly feasible.

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Jan 21, 2014

@hswolff I've thought about that too, the issue with that being who watches the watchdog? Sure we can kick off the actual server in a child process but we still need something to then watch the parent process. As well you'd still probably want the parent process running as a daemon/service so you'd just net yourself another layer -- is there some benefit there I'm missing?

@hswolff

This comment has been minimized.

Copy link
Member

commented Jan 21, 2014

Having the 'watchdog' server start the Ghost server gives it the ability to start and stop the Ghost server.

So we can emit a message from Ghost which the parent process watches for to know when to restart the Ghost server.

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Jan 21, 2014

@hswolff Sure, but what if part of the update was to the watchdog?

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Jan 21, 2014

Being able to restart Ghost at the end of an update is a nice to have. One way that I thought we might be able make to it possible is to make it possible to tell his how to restart Ghost via config.js.

The default is that we can't, but, if say you're using forever, you'd put something like:

restartScript: forever restart index.js

This is similar to what we put in package.json, telling npm how it can run and test Ghost. I'm not sure how well it would work, but it's an idea?

As for the CLI tools, I am not suggesting that a CLI tool should replace the UI. Rather that we should code the update tools in such a way that they can be run from a UI or from the CLI. Some users may prefer the CLI, especially if they are running multiple blogs.

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Mar 1, 2014

So I've given this some more thought and the below is what I'm thinking as far as process. I've bolded steps that would be additions to the current PR and italicized anything that I think would fall into the "nice to have but not right now" bucket.

  1. Check for write privileges against Ghost root using fs.stat and checking the mode property. Fail if we don't have write privileges.
  2. Check for update against new web service. Fail if no update is found.
  3. Download the zip to /content/upgrade/.
  4. Extract the contents to /content/upgrade/.
  5. Unlink the zip file.
  6. Would be nice to flip Ghost into a "maintenance" mode at this point sometime down the road, but I don't think this is a must have.
  7. Backup the database to the usual location.
  8. Copy the new files on top of the current installation.
  9. Invoke npm install and npm update the same way we do for apps.
  10. Unlink /content/upgrade/.
  11. We would flip off "maintenance" mode at this point.
  12. Unlink old/deprecated files. The list of these files would be stored in a .json file and we would run a migration style process to delete files from v.Current up through v.Update.
  13. Create a persistent notification requesting the user to restart the service.

Note: Regarding avoiding restarting the process, I did eventually find a way to clear the require cache. It is actually pretty simple looking, but is not full proof. You basically do delete require.cache[moduleName];. The issues with that are: you lose any state that may have been maintained by that module instance and if there were other references those references may not be GC'd. Therefore I would personally not recommend going down this route, but we could play with it later.

@javorszky

This comment has been minimized.

Copy link
Member

commented Mar 1, 2014

Hm, what happens if npm install and npm update fails? Especiall now that npm is having cert issues lately.

@gotdibbs

This comment has been minimized.

Copy link
Member

commented Mar 1, 2014

@javorszky That is a very good question. My guess would be that we either halt the process with a relevant error, or continue and display the error at the end. I don't see a great reason for one over the other. The same thing could happen if you did the upgrade manually and npm was down.

@hswolff

This comment has been minimized.

Copy link
Member

commented Mar 3, 2014

julianlam from IRC just posted this excellent blog post on how they took cues from nodemon on how to handle restarting their application. Seems like it's definitely a viable solution.

http://blog.nodebb.org/restarting-node-applications/

@julianlam

This comment has been minimized.

Copy link

commented Mar 3, 2014

The full loader can be found here 😄

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Mar 4, 2014

Lets make a separate issue for doing the restart? I think it requires some significant and potentially interesting changes to Ghost.

From @gotdibbs's list above, I'm most interested in making sure 1,2 and 13 happen and probably 10 as well whilst we're there.

9 can be done in a separate issue, unless @gotdibbs thinks it's easy to add it in without code duplication.

12 can be left for now, as for version 2 of the updater I'd like to do a full diff between the two directories in step 8, copy new & modified files only, and delete removed ones, without having to store additional meta data. But that's very version 2 and can be ignored for now.

@ErisDS ErisDS modified the milestone: 0.6 Apps Sep 2, 2014

@ErisDS

This comment has been minimized.

Copy link
Member Author

commented Oct 8, 2015

Closing this as it has no traction at the moment. Long term I'm sure we'll revisit some form of upgrade tool.

@ErisDS ErisDS closed this Oct 8, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.