Rails 4 Asset Caching Doesn't Clean Old Assets #123

Closed
hone opened this Issue Jul 31, 2013 · 41 comments

Comments

Projects
None yet
Owner

hone commented Jul 31, 2013

Since we started caching assets in Rails 4. There's an issue that old assets aren't getting cleared. We need some way of cleaning up old assets that aren't being used anymore.

Update: To clarify, the issue is that there are old assets that are deleted that still linger.

If needed, I can keep that app frozen on Heroku so it can be looked at by staff.

Owner

hone commented Jul 31, 2013

@matthewmcmillion I think we have a fix in mind. Can you fork it and provide the app name of the fork?

Yep, you can use 'bitpivot-web'.

Contributor

schneems commented Jul 31, 2013

I can confirm that Rails4 does clean assets on heroku. However it takes 3 changes to that file for sprockets to clear out the files.

Steps, run this 4 times and compare output:

echo "body {background-color: red}">> app/assets/stylesheets/application.css
git add .; git commit -m "assets changed 1"
git push heroku master
heroku run ls public/assets | awk /application/

Closing this issue.

schneems closed this Jul 31, 2013

The problem isn't with changed files, it's with deleted files. The files were removed 20 or 30 commits ago, but still end up in asset precompile output.

Contributor

schneems commented Jul 31, 2013

Ill check that case. Thanks for the info.

schneems reopened this Jul 31, 2013

Contributor

schneems commented Aug 1, 2013

I can confirm that rake assets:clean does not remove deleted assets which is somewhat unexpected. One of their goals is to make zero downtime deploys easier by keeping around older versions of assets so if your webpage loads with old asset urls and your asset requests hit a new server everything will still work.

This isn't an issue with Heroku but rather sprockets and sprockets-rails which provides those rake tasks. Seems bad that deleted assets never get removed and will continue to grow indefinitely. A read through cache would eventually fix this issue on our end.

If you want to push a file out of cache for security reasons, I would recommend creating a new blank file with the same name, then modify it by adding whitespace and push three times. It's not ideal but it's better than nothing.

I can also confirm that rake assets:clobber doesn't clear them either.

mcmillion referenced this issue in rails/sprockets-rails Aug 1, 2013

Closed

Allow deleted assets to be removed from cache #79

@schneems schneems added a commit to schneems/sprockets that referenced this issue Aug 5, 2013

@schneems schneems Remove Deleted Assets On Clean
Right now sprockets will preserve the last number copies of an asset after it is modified, lets call this `keep`. If clean is called `X` more times than `keep` is specified and there are more than `keep` copies, then `X` backups will be removed from the file system. This behavior has two desirable behaviors: allowing for rolling deploys since we may need to render old assets from the new server on deploy, and preventing the compiled asset directory from growing continuously.

Right now if a user deletes an asset, it will not be removed from the compiled directory regardless of how many times compile or clean are called. If a developer needs to delete files frequently, this means that the compiled directory will only continue to grow in size even if clean is being called. A user may need to delete an asset due to security concerns: 
heroku/heroku-buildpack-ruby#123 and it is currently impossible without resorting to a hack.

This PR allows sprockets to remove files from the compiled directory that have been deleted from the source directory. To do this we need to tag each asset with a `compiled_at` time and store a global `compiled_at` time to know the last time compile was run. Instead of being deleted instantaneously, removed assets are stored in a `deleted_assets` key in the top level manifest. Every time `clean` is run it will increment a `generation` in the asset. When `generation` == `keep` then the asset will be removed from the project. This allows us to preserve rolling deploy functionality for deleted assets, while not allowing the deleted assets to continue to take up an ever increasing amount of space.

ATP
bcc5e76

@schneems schneems added a commit to schneems/sprockets that referenced this issue Aug 7, 2013

@schneems schneems Remove Deleted Assets On Clean
Right now sprockets will preserve the last number copies of an asset after it is modified, lets call this `keep`. If clean is called `X` more times than `keep` is specified and there are more than `keep` copies, then `X` backups will be removed from the file system. This behavior has two desirable behaviors: allowing for rolling deploys since we may need to render old assets from the new server on deploy, and preventing the compiled asset directory from growing continuously.

Right now if a user deletes an asset, it will not be removed from the compiled directory regardless of how many times compile or clean are called. If a developer needs to delete files frequently, this means that the compiled directory will only continue to grow in size even if clean is being called. A user may need to delete an asset due to security concerns: 
heroku/heroku-buildpack-ruby#123 and it is currently impossible without resorting to a hack.

This PR allows sprockets to remove files from the compiled directory that have been deleted from the source directory. To do this we need to tag each asset with a `compiled_at` time and store a global `compiled_at` time to know the last time compile was run. Instead of being deleted instantaneously, removed assets are stored in a `deleted_assets` key in the top level manifest. Every time `clean` is run it will increment a `generation` in the asset. When `generation` == `keep` then the asset will be removed from the project. This allows us to preserve rolling deploy functionality for deleted assets, while not allowing the deleted assets to continue to take up an ever increasing amount of space.

ATP
5472f35

schneems referenced this issue in sstephenson/sprockets Aug 8, 2013

Closed

Remove Deleted Assets On Clean #464

@schneems schneems added a commit to schneems/sprockets that referenced this issue Aug 8, 2013

@schneems schneems Remove Deleted Assets On Clean
Right now sprockets will preserve the last number copies of an asset after it is modified, lets call this `keep`. If clean is called `X` more times than `keep` is specified and there are more than `keep` copies, then `X` backups will be removed from the file system. This behavior has two desirable behaviors: allowing for rolling deploys since we may need to render old assets from the new server on deploy, and preventing the compiled asset directory from growing continuously.

Right now if a user deletes an asset, it will not be removed from the compiled directory regardless of how many times compile or clean are called. If a developer needs to delete files frequently, this means that the compiled directory will only continue to grow in size even if clean is being called. A user may need to delete an asset due to security concerns: 
heroku/heroku-buildpack-ruby#123 and it is currently impossible without resorting to a hack.

This PR allows sprockets to remove files from the compiled directory that have been deleted from the source directory. To do this we need to tag each asset with a `compiled_at` time and store a global `compiled_at` time to know the last time compile was run. Instead of being deleted instantaneously, removed assets are stored in a `deleted_assets` key in the top level manifest. Every time `clean` is run it will increment a `generation` in the asset. When `generation` == `keep` then the asset will be removed from the project. This allows us to preserve rolling deploy functionality for deleted assets, while not allowing the deleted assets to continue to take up an ever increasing amount of space.

ATP
59a0fb6

marnen commented Aug 26, 2013

I'm having an issue that looks like this. At least in my case, what's going on is that the assets Rake tasks think that the assets directory is /app/public/assets, so they're not touching the files in /app/assets. This also means that the assets don't get compiled into the right place...

@hone Sup!

Any status on this issue? Is there a good workaround at the moment to delete the old assets? My app is http://enroll-staging.herokuapp.com and I'm pretty sure the wonky assets behavior I'm getting are related to this issue.

Any way to definitely test to tell if this is affecting my app?

Owner

hone commented Sep 6, 2013

hey @jessmartin! We need to catch up sometime.

I don't have any news here yet. This issue is basically there are assets that you delete that are still available b/c rake assets:clean does not remove them. It does keep up to 3 versions of your assets around. I'm not sure what wonky behavior you're talking about though.

Terence - found the problem and it wasn't related.

And we should definitely catch up sometime!

On Fri, Sep 6, 2013 at 4:06 PM, Terence Lee
<notifications@github.com<javascript:_e({}, 'cvml',
'notifications@github.com');>

wrote:

hey @jessmartin https://github.com/jessmartin! We need to catch up
sometime.

I don't have any news here yet. This issue is basically there are assets
that you delete that are still available b/c rake assets:clean does not
remove them. It does keep up to 3 versions of your assets around. I'm not
sure what wonky behavior you're talking about though.


Reply to this email directly or view it on GitHubhttps://github.com/heroku/heroku-buildpack-ruby/issues/123#issuecomment-23965762
.

Jess Martin
http://jessmart.in

Jess Martin
http://jessmart.in

r38y commented Nov 8, 2013

@jessmartin what was the issue? I'm encountering something similar but it looks like the manifest ends up being wrong so my pages reference the wrong compiled file.

@r38y I'm wracking my brain, but honestly I can't remember. I even went back and looked at my commits from that day - nothing. I'm pretty sure I had something configured incorrectly on my end. It didn't turn out to be a heroku problem.

r38y commented Nov 8, 2013

@jessmartin thanks for trying. After hours of banging my head against the wall, I figured out it was a single file "manifest.json" that was generated a while back when we tried the first (non-complete) rake task at the top of https://gist.github.com/eric1234/5692456.

Dakuan commented Jan 22, 2014

I am also being affected by this, an old javascript file that should no longer exist is still hanging around and is causing bugs.

Contributor

schneems commented Jan 22, 2014

Try https://github.com/schneems/sprockets_better_errors

It's likely a missing asset dependency declaration.

Sent from Mailbox for iPhone

On Wed, Jan 22, 2014 at 8:45 AM, Dominic Barker notifications@github.com
wrote:

I am also being affected by this, an old javascript file that should no longer exist is still hanging around and is causing bugs.

Reply to this email directly or view it on GitHub:
#123 (comment)

andybry commented May 8, 2014

I was also affected by this issue. I was able to resolve it (i.e. uncache all my assets on Heroku) by incrementing the config.assets.version variable in ${project-root}/config/application.rb

This causes all the URL hashes to be modified, and means that no old files are included in the application.

@andybry would you mind elaborating on this a bit more, perhaps provide an example of what you done? I'm having same issue with old assets being uploaded, would appreciate anything you can contribute, thank you

andybry commented May 13, 2014

@richlewis14 I can't show you the actual example because its in proprietary code owned by the company I work for, but here's an example of what I meant in a bare bones Rails 4 application: andybry/assets-increment-example@5fad722 and then push to Heroku. Any dramatic changes to the assets should further increment the number in that string to prevent similar issues.

I've seen some posts on the Web that claim that you only need to use that configuration variable when upgrading from Rails 3 to 4, but nonetheless it did work for me (I think sprockets uses it when it's generating the hashes it uses in caching compiled files and in URLs, so it just causes everything to uncache). Perhaps Heroku is another use case that its suited to.

Hope that's helpful.

oh i see, you are just manually changing the version number..I thought you had a script that auto increments on each deployment... thanks for the example though

well thats a shame, it did not work for me, still uploaded a lot of images that have been deleted

I'm also suffering from an outdated version of an asset included in compiled output.

agrberg commented Aug 15, 2014

I'm having the same problem in a different way. We're using haml_coffee_assets and changed some config settings in config/application.rb. The templates do not get rebuilt until they are changed. In addition, only the templates changed get recompiled, the rest remain cached.

Contributor

schneems commented Aug 15, 2014

@agrberg that's a different issue. Likely sstephenson/sprockets#534 you need to use depend_on and depend_on_asset in your templates. Also your asset files must change for sprockets to re-build them.

agrberg commented Aug 15, 2014

I was also able to fix it using the heroku-repo plugin: https://github.com/heroku/heroku-repo.

It's possible that Heroku made some changes/fixes on their end. I'd been having trouble with asset compilation for months, and then yesterday i mistakenly deployed without my precompiled assets and.. everything just worked like it was supposed to again.

dabit commented Sep 18, 2014

Any chance someone found a solution for this problem? This is what public/assets looks like for ONE FILE.

    2014-07-24 00:13 all-43bbe8085c6a56f45360dbc3c39c8595.js
    2014-07-24 14:14 all-1928d1124df3eb2279d2e282fb7fb633.js
    2014-07-24 18:53 all-f14749f12c598c01232b4494a160ba4b.js
    2014-07-26 22:51 all-f85e9276b7e16b621eb8d95197602f47.js
    2014-07-27 00:55 all-c6317063b7e2d91f0febbb0ee4abe263.js
    2014-07-30 00:02 all-136ba95b5fd074796295d040a943d9fd.js
    2014-07-30 18:34 all-1aad328ad71053bf1de4a01636bd6c83.js
    2014-07-30 19:25 all-418093e63c83d79763889f64393ad0cc.js
    2014-07-31 00:04 all-c1bbdfcf8cdb1320b470857fc2723966.js
    2014-07-31 02:17 all-22afea8de8efe3bc0ef07d4df28cedeb.js
    2014-07-31 03:13 all-6052f4e7d7e3e50bfcafe3d6bea53748.js
    2014-07-31 17:50 all-4321e4ae38c4d7aaf774f1032ea7a9ba.js
    2014-07-31 20:09 all-6414c4b10ea8e9ee2b55912247af4f3b.js
    2014-08-02 11:37 all-4c562b0a144c44bc60da29546d1700e2.js
    2014-08-04 19:29 all-76cf73ea8dc082ceaa4a7747f1dd24b8.js
    2014-08-05 14:50 all-e70c698554cabc18fe460cca4a66c6d4.js
    2014-08-05 17:43 all-87aa5cea5cbe07c7442dddccce95f636.js
    2014-08-06 17:05 all-1dcf6373d22d1e3ef83ae14aaefa05de.js
    2014-08-07 14:01 all-a0a69f595750cbabe7968129d55e2fc2.js
    2014-08-07 19:17 all-d9e1ee47852cb0d89f9e89e673d245c5.js
    2014-08-08 12:03 all-3b06eb690940e37babd530019c7def8b.js
    2014-08-11 11:30 all-d84bb5c4d529ca502771dd04818d206f.js
    2014-08-11 12:33 all-46fbbe93c25e53020f62f786c03f45ac.js
    2014-08-11 14:14 all-725525d65681e3f942b72e0da989a424.js
    2014-08-11 19:00 all-9551c0fc796c630e8b867fddfb9b2835.js
    2014-08-21 13:38 all-2c27f7e798c79bf79b8595b54ee02a9d.js
    2014-08-21 18:59 all-87faf852267ee0d3fa58001206a4e80d.js
    2014-08-22 11:44 all-0938327d0826899b9f6b9f11761416e6.js
    2014-08-27 11:29 all-1cae4421e40d6f5e6ee9082918e5fb7c.js
    2014-08-27 18:22 all-586792c677469b386230d85a9397a601.js
    2014-08-27 20:51 all-1856603ec44d5e25b4414d7fc52a896e.js
    2014-09-01 14:08 all-ebc5022b5347a7f701182ea4ad1f4607.js
    2014-09-02 14:10 all-ca9863a9a865be73a9e5bd1f2d590e7e.js
    2014-09-10 11:28 all-203c46ae496ee517f90eb99e6df73f5d.js
    2014-09-10 13:47 all-e64b3ff2ca1e552186169ff6444a8fd7.js
    2014-09-17 14:56 all-0eb6e54cd57b8acafc5d573f0804dd8f.js

As you can see I have a copy of my assets that are 2 months old and it has now become a very big problem because this is making my app hit the 300MB slug size limit. I can't deploy anymore.

@schneems or @hone Is there any way I can just manually erase public/assets during a deploy?

Contributor

schneems commented Sep 19, 2014

You're probably seeing this issue: rails/sprockets-rails#138

Upgrade to the master of sprockets-rails to fix the root issue.

To clear your cache, you can use the heroku repo plugin https://github.com/heroku/heroku-repo

1lyan commented Oct 2, 2014

Hi, @dabit. We solved the problem the following way:

What i did :

  • repo plugin (reset + purge_cache)
  • update config.assets.version
  • push

And it worked

Contributor

schneems commented Oct 17, 2014

And it worked

And you'll have to do that EVERY time you deploy unless you fix the underlying problem...

Update your sprockets-rails dependency please

mhoran commented Oct 20, 2014

We noticed that despite setting our assets:clean task to retain only a single copy of assets, the buildpack asset cache would restore deleted assets on subsequent deploys. See #300 for details and a fix for the buildpack.

schneems closed this Oct 20, 2014

Contributor

schneems commented Oct 20, 2014

The Issue

To clarify this is a known issue (rails/sprockets-rails#138), you can solve it by doing this:

The Fix

Upgrade to the latest sprockets-rails to fix the root issue. To clear your cache manually, you can use the heroku repo plugin https://github.com/heroku/heroku-repo. If your "fix" involves rev-ing the version constantly or constantly using the repo plugin, it's not really a fix.

If you have a different issue, or this does not solve your problem please create a new issue. Adding onto this one obscures the fix.

modosc commented Oct 23, 2014

has anyone else managed to upgrade to sprockets-rails master per @schneems comment? we're on rails 4.1.6 and it didn't seem possible since rails 4.16 depends on sprockets-rails ~> 2.0.

are there specific changes we could cherry-pick out?

Confirm that using https://github.com/heroku/heroku-repo to purge assets works as a fix for this.

Contributor

schneems commented Dec 1, 2014

It was master at the time of that comment. I believe that commit it is released now.


Sent from Mailbox

On Mon, Dec 1, 2014 at 10:11 PM, Andrew Cockerham
notifications@github.com wrote:

Confirm that using https://github.com/heroku/heroku-repo to purge assets works as a fix for this.

Reply to this email directly or view it on GitHub:
#123 (comment)

sprocket-rails has the mentioned fix in 2.1.4.

Changelog:
https://github.com/rails/sprockets-rails/blob/master/CHANGELOG.md#214

Pull request:
rails/sprockets-rails#140

lneffa referenced this issue in vigetlabs/gulp-rails-pipeline Mar 7, 2015

Closed

Rails asset caching #6

Thanks for the fix. I had a similar issue that was making me tear my hair out, the heroku-repo tool fixed it :)

I was experiencing similar issue where heroku was serving the old js file 10 commits ago. After precompiling the assets locally then push to heroku finally fixed it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment