Files that already exist on S3 are still uploaded #16

Closed
jasonhutchens opened this Issue Nov 8, 2011 · 10 comments

Projects

None yet

3 participants

@jasonhutchens

We have many more than 1000 files in our S3 bucket, but the default value for max-keys in fog is 1000. This means that AssetSync::Storage::get_remote_files only returns the first 1000 files found, meaning that, although local_files_to_upload should be empty, it ends up containing many thousands of files.

I believe that the same bug means that requesting delete instead of keep will only delete the first 1000 files.

Contributor

Is this a bucket with 1000 static assets in it?

Are they all used in your HTML / CSS and JS build ?

yes, we've got more than 1000 static assets in the bucket, partly because we're not deleting old ones and partly because more are being uploaded than necessary; it looks like Fog only returns the first 1000 no matter what; I've forked and am playing around with a fix

Contributor

Thanks! Version 0.1.10 should solve this. Let me know if there are any further issues.

@davidjrice davidjrice closed this Nov 15, 2011
scottkf commented Jan 6, 2012

It seems like I still have this issue, without about 3000 items in the bucket. In a rails app, the app.js and app.css are processed last, and don't get changed.

Contributor

@scottkf really? Are you on the latest version (0.2.4). I've had no problems like this.

It is likely still related to Fog.

If you want to put together a test case or give us access to a basic rails app that exposes this problem I'd be glad to give you a hand. Not the easiest one to replicate...

scottkf commented Jan 16, 2012

@davidjrice I'm using 0.2.4, I haven't had time to put a test case yet, but I'll let you know, thanks. What's weird though, is that if I choose to use the manifest.yml and limit the amount of files, the files listed still don't get updated.

scottkf commented Jan 18, 2012

@davidjrice I still didn't figure out why it wasn't working, but I deleted application.js and application.css, compiled the assets again and it was re-uploaded, and now syncs properly.

Contributor

Great. Well definitely upgrade to 0.2.5 as well to be sure. There was an issue with 0.2.4

On Wednesday, 18 January 2012 at 20:32, Scott Tesoriere wrote:

@davidjrice I still didn't figure out why it wasn't working, but I deleted application.js and application.css, compiled the assets again and it was re-uploaded, and now syncs properly.


Reply to this email directly or view it on GitHub:
#16 (comment)

@jasukkas jasukkas pushed a commit to AdsOptimal/asset_sync that referenced this issue Sep 15, 2016
Jason Hutchens fixes AssetSync/asset_sync#16 e8c5307
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment