Navigation Menu

Skip to content

Commit

Permalink
FIX: prefix should precede folder path (follow-up on 10fbb07)
Browse files Browse the repository at this point in the history
  • Loading branch information
@rishabhnambiar
rishabhnambiar committed Jan 15, 2019
1 parent 59e5af4 commit ff8f9dc
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion lib/tasks/uploads.rake
View file
Expand Up @@ -275,7 +275,7 @@ def migrate_to_s3

s3_objects = []
prefix = Rails.configuration.multisite ? "#{db}/original/" : "original/"
options = { bucket: bucket, prefix: prefix + folder }
options = { bucket: bucket, prefix: folder + prefix }

loop do
response = s3.list_objects_v2(options)
Expand Down

3 comments on commit ff8f9dc

@rishabhnambiar
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The listing issue is fixed for both kinds of bucket paths ⬇️

Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files => 64 files
 - Listing S3 files. => 64 files
 - Syncing files to S3

@ZogStriP could I have a sanity check on this output? Especially the remap and replace destinations?

[details=without folder path]
DRY_RUN=1
DISCOURSE_S3_REGION="us-east-1"
DISCOURSE_S3_BUCKET="rishabhmultisite3"
DISCOURSE_S3_CDN_URL="https://discourse-cdn-aws1.com/pocketgems"
rake uploads:migrate_to_s3

****************************** DRY RUN ******************************
Migrating uploads to S3 for 'default'...
Uploading files to S3...

  • Listing local files => 64 files
  • Listing S3 files. => 0 files
  • Syncing files to S3

uploads/default/original/1X/7dc054e635cfb985d8d790584fd14996fed12eae.jpeg => original/1X/7dc054e635cfb985d8d790584fd14996fed12eae.jpeg

.....

.....

uploads/default/original/1X/29e24a112e295db84e91826862da2fccb96d0df3.png => original/1X/29e24a112e295db84e91826862da2fccb96d0df3.png

Updating the URLs in the database...
REPLACING '/uploads/default/original/(\dX/(?:[a-f0-9]/)[a-f0-9]{40}[a-z0-9.])' WITH //rishabhmultisite3.s3.dualstack.us-east-1.amazonaws.com/original/\1

REMAPPING http://localhost//rishabhmultisite3.s3.dualstack.us-east-1.amazonaws.com TO https://discourse-cdn-aws1.com/pocketgems
Done!
[/details]

[details=with folder path]
DISCOURSE_S3_REGION="us-east-1"
DISCOURSE_S3_BUCKET="rishabhmultisite3/cloud"
DISCOURSE_S3_CDN_URL="https://discourse-cdn-aws1.com/pocketgems"
rake uploads:migrate_to_s3

****************************** DRY RUN ******************************
Migrating uploads to S3 for 'default'...
Uploading files to S3...

  • Listing local files => 64 files
  • Listing S3 files. => 0 files
  • Syncing files to S3

uploads/default/original/1X/7dc054e635cfb985d8d790584fd14996fed12eae.jpeg => cloud/original/1X/7dc054e635cfb985d8d790584fd14996fed12eae.jpeg

.......

uploads/default/original/1X/21e6700c21f70f299af6c4e74048c9e9ccf32983.png => cloud/original/1X/21e6700c21f70f299af6c4e74048c9e9ccf32983.png

Updating the URLs in the database...
REPLACING 1/uploads/default/original/(\dX/(?:[a-f0-9]/)[a-f0-9]{40}[a-z0-9.]) WITH //rishabhmultisite3.s3.dualstack.us-east-1.amazonaws.com/cloud/original/\1

REMAPPING http://localhost//rishabhmultisite3.s3.dualstack.us-east-1.amazonaws.com/cloud TO https://discourse-cdn-aws1.com/pocketgems
Done!

[/details]

@ZogStriP
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from a cursory look. Have you tested multisite?

@rishabhnambiar
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did so by manually setting prefix to use "#{db}/original/" on my test install.

[details=without folder path using a multisite prefix]

uploads/default/original/1X/b92f9ef1eb09102e8c48fb0c1d872c82c9fcbdff.png => default/original/1X/b92f9ef1eb09102e8c48fb0c1d872c82c9fcbdff.png
uploads/default/original/1X/21e6700c21f70f299af6c4e74048c9e9ccf32983.png => default/original/1X/21e6700c21f70f299af6c4e74048c9e9ccf32983.png

Updating the URLs in the database...
REPLACING '/uploads/default/original/(\dX/(?:[a-f0-9]/)*[a-f0-9]{40}[a-z0-9\.]*)' WITH '//rishabhmultisite10.s3.dualstack.us-east-1.amazonaws.com/default/original/\1'
REMAPPING 'http://localhost//rishabhmultisite10.s3.dualstack.us-east-1.amazonaws.com' TO 'https://discourse-cdn-aws1.com/pocketgems'

[/details]

[details=with folder path using a multisite prefix]

uploads/default/original/1X/b92f9ef1eb09102e8c48fb0c1d872c82c9fcbdff.png => cloud-file-uploads/default/original/1X/b92f9ef1eb09102e8c48fb0c1d872c82c9fcbdff.png
uploads/default/original/1X/21e6700c21f70f299af6c4e74048c9e9ccf32983.png => cloud-file-uploads/default/original/1X/21e6700c21f70f299af6c4e74048c9e9ccf32983.png

Updating the URLs in the database...
REPLACING '/uploads/default/original/(\dX/(?:[a-f0-9]/)*[a-f0-9]{40}[a-z0-9\.]*)' WITH '//rishabhmultisite10.s3.dualstack.us-east-1.amazonaws.com/cloud-file-uploads/default/original/\1'
REMAPPING 'http://localhost//rishabhmultisite10.s3.dualstack.us-east-1.amazonaws.com/cloud-file-uploads' TO 'https://discourse-cdn-aws1.com/pocketgems'

[/details]

Please sign in to comment.