Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

rollback of cap deploy:assets uses newest assets #210

Closed
pwim opened this Issue · 12 comments

8 participants

@pwim

Rails uses manifest.yml to determine which assets to use. Because the symlink task links the public assets to the shared one, the manifest.yml is overwritten whenever we deploy. Thus, even if we rollback, Rails still points to the old assets.

@leehambley
Owner

@pwim I'm happy to take a patch for that, it's not something I'm likely to be able to make time to fix in the next 2-4 months. (we're focusing on major rewrites of Cap)

I'd say that relying on rollbacks is bad form, but without the ability to test deploys, I know that happens.

The asset pipeline support was contributed by @cgriego, and I've never really looked further than that into it (I'm not using it myself) Hopefully now that I've /cc'ed @cgriego he'll offer something useful.

One query, isn't the point of the asset pipeline that assets from /all/ deploys are kept, so surely having extra (from the failed deploy) assets hanging around, isn't that much of a problem?

@pwim

@leehambley Thanks for quick response. Sorry that I wasn't clear. Keeping around assets is fine, it's just that the manifest file references the wrong set of assets. So in my case, I had deleted some css definitions from the new version (as they were no longer used in that version of the markup), but when I rolled back, Rails still pointed to the new version of the css, so some elements weren't styled properly.

I'd say that relying on rollbacks is bad form, but without the ability to test deploys, I know that happens.

Do you have some recommendation of what we should do instead?

@leehambley
Owner

Let's wait for @cgriego to weigh in with some commentary on how he's dealing with this (if he's dealing with it) - at any rate you'll be able to patch the offending method, but I can't recommend what to do as I'm not familiar with the asset pipeline.

@jacobat

Right now Capistrano recompiles assets on every deploy. So a simple solution is to not symlink the assets folder into shared. I'm not sure what the reason for doing the shared folder in the first place was, so maybe I'm missing something.

@jacobat

The reasoning was right in the description of the symlink task:

Assets are shared across deploys to avoid mid-deploy mismatches between old application html asking for assets and getting a 404 file not found error.

@cgriego

Sorry guys, my laptop is in the shop so it's taking me longer to respond to things.

We haven't run into this issue since we don't end up rolling back very often. The assets recipe implements the deployment guidelines that came from the Rails guides and DHH's RailsConf keynote on the pipeline. The shared assets across deploys guideline, I think, came exclusively from the keynote. Sharing assets allows seamless continuous deploys that the user will not have to know went out in the middle of them using the site. However, rollbacks not working correctly is pretty egregious. The manifest file was a late addition to the pipeline, an optimization I think. The people who added it probably was not considering the seamless deployment use case. There's a couple of options here:

  1. Recompile assets on rollback. This fixes rollback but makes rolling back quite slow, which violates the expectation of rolling back with Capistrano. (Should just be a symlink change and a restart.)
  2. Remove the symlink and stop sharing assets across deploys. This fixes rollback and allows for fast rollbacks.
  3. Configure Rails to move the manifest file. It looks like the app.config.assets.manifest config option allows the manifest file to be moved outside of the assets folder, that way it can be specific to a release, the assets can be shared across deploys allowing for seamless deploys, rollback works, and rollback if fast. The downside here is needing config changed in the application for everything to work smoothly on rollback. Given how long this feature has been out and this being the first report of the issue, adding documentation on the needed config might be enough.

@pwim Are you willing to apply the configuration change to your application and see that it makes your deployments and rollbacks work as expected? Setting it to Rails.root.join("config") should work.

@pwim

@cgriego We thought about modifying app.config.assets.manifest, but we ultimately resolved it by removing the symlink task. We're using cloudfront to serve our assets, so there is no issue with continuous deploys. As I assume it's standard practice to use a CDN for serving assets, maybe it is better behaviour to remove the symlink task?

@holli

We ran across this same problem. Quite annoying feature in sense that it takes time to figure out where is the problem. And during a rollback we are in hurry anyways.

@cgriego's third option seems to work ok. Of course the third option has some problems. If you are linking directly to any files without the end-hash, e.g. by manually having <script src="/assets/application.js"/>, then changing the manifest file won't help as those asset-files are always replaced by newer ones. If you are using assets correctly <script src="<%= asset_path ... %> "/> then this option is enough.

Sharing assets (symlinking) works well. Without asset sharing we could have some minor js loading problems every time we would deploy or rollback. At least amazon's CDN fetches the assets from our main server so the CDN won't really affect the original problem.

+1 for adding the third option to documentation.

Quick one to some-one who is searching this, add following:

production.rb >

config.assets.manifest = Rails.root.join("config") # cap deploy:rollback with assets (https://github.com/capistrano/capistrano/issues/210)
@ndbroadbent

I think it would be a good idea for the assets recipe to set config.assets.manifest = Rails.root.join("config") by default, to solve the rollback problem for everyone.

I've been working on a way to speed up asset compilation, and I initially added some behaviour to delete old assets after each compile. I didn't think of the case where cached html would request old assets.
In light of that, what do you think about an option like clear_assets_older_than, with a default value of 5.days.ago? Old assets could then be deleted based on their modification time.

I'm thinking that asset clearing is probably best as part of the assets capistrano recipe, instead of in the asset:precompile task. What do you think?

@jrochkind

It seems in general rollbacks are kind of tricky. That there are some somewhat significant rollback bugs that nobody has noticed says that people don't use it a lot. Also because people don't use it a lot, it's quite possible there's a bug in the rollback code, not just for assets but for other things including third party plugins you might use. It's a bit tricky to get rollbacks right when writing any recipe.

@leehambley even says "I'd say that relying on rollbacks is bad form".

Another option to relying on rollbacks to is to make sure to tag every release, and then when you want to rollback, simply deploy the penultimate (or older) tag. It's not going to be as quick as a rollback, it's a complete deploy, but it should be just as reliable as doing any other cap deploy. If you are interested in that, it's one of the use-cases of my https://github.com/jrochkind/cap_git_tools cap plugin. Which I don't think has been getting a lot of attention/use, so my plugin might have bugs in it too (it's a bit tricky to get git integration working reliably too), but to me git tagging integration seems like a no-brainer, and I'm happy to take pull requests or add committers if you are interested in the approach I'm taking there.

@ndbroadbent

Hi everyone, I've had a proper go at solving this issue. Please take a look at #281 and let me know your thoughts. Note that your manifest.yml file needs to be written to the standard public/assets/ in order for everything to work properly.

Cheers!

@carsomyr

All, please move all discussion to Pull Request #281. Thanks!

@carsomyr carsomyr closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.