Export user photos #5685

margori · 2015-02-19T03:20:59Z

My solution for bug #1802

Implemented "Export photos" button in user settings, currently disabled.

I'm using 'zip' files instead of 'tar', 'cause it's widely known and easily handled.

jhass · 2015-02-19T04:08:37Z

I fear there's a lot more to consider to this. Let's start with some assorted points that immediately pop into my mind.

We shouldn't do such an expensive (and thus potentially lengthy) operation directly in the action, it should be a background job, else it will fail and run into timeouts.
- This then raises the question of how we inform the user that the download is ready, when mail is enabled, we could just sent a mail with the download link.
- However if mail is not available, we need a flow for that. One imaginable scenario is to pop a message to the user that the export will be prepared and they should check back in a few minutes. This then means we should keep the last export for a configurable amount of time.
  - That means we would need a cleanup job for the exports.
  - An existing export would need to be cleaned out if the user requests a new one.
  - A download link for an existing export would need to be displayed.
- The download URL would need to contain a token, so they're not guessable.
- We would need to store the URL somewhere in the DB
For the unlikely event that I can be convinced to not put this into a background job, we should consider and investigate X-Sendfile as delivery mechanism.
We probably should investigate and verify whether the chosen approach works on a PaaS platform such as Heroku and whether we want to support storage locations like Amazon S3.

Flaburgan · 2015-02-19T10:09:21Z

You can have a look at how this is handled by #5499

jhass · 2015-02-19T12:02:59Z

#5499 at least backgrounds the work and sends a mail, it doesn't have an optimal UI flow either yet nor any cleanup.

goobertron · 2015-02-19T15:16:12Z

However if mail is not available, we need a flow for that. One imaginable scenario is to pop a message to the user that the export will be prepared and they should check back in a few minutes. This then means we should keep the last export for a configurable amount of time.

How about: an alert that the job may take some time, and then the user gets a notification in the usual way when the export is ready, with a download link. The export could be kept until downloaded (perhaps up to a maximum time).

margori · 2015-02-19T17:23:14Z

I'm gonna try to create a output stream.

If can't, I'm gonna use #5499 method so user see same behavior for both exports.

margori · 2015-02-19T21:02:01Z

Best approach to stream a zip file is using https://github.com/fringd/zipline but it's not ported to rails 4 yet. :-(

margori · 2015-02-20T19:10:06Z

Done: used #5499 way of export data.

Comments are very welcome.

jhass · 2015-02-20T19:13:52Z

app/models/user.rb

+
+    Zip::ZipFile.open(export.path, Zip::File::CREATE) do |zipfile|
+      photos.each do |photo|
+          photo_path = "#{Rails.root}/public/uploads/images/#{photo.remote_photo_name}"


We can't assume that, it might be hosted on S3.

If we can't then PR #5499 requires review as well.

Oh! You're talking about line 331. I thought you were talking about line 317.

I assume that files are stored in same machine as web server. :-(

margori · 2015-02-23T03:41:08Z

New try. In this opportunity I'm assuming files could not be in local file system.

jhass · 2015-02-23T11:51:08Z

app/models/user.rb

+      photos.each do |photo|
+        temp_photo = Tempfile.new(photo.remote_photo_name, encoding: 'ascii-8bit')
+        begin
+          photo_path = "#{Rails.root}/public/uploads/images/#{photo.remote_photo_name}"


Sorry, but Rails.root does not give you the upload location or even the sites domain or anything, it gives you the absolute path on the filesystem where the application is located, so this still doesn't work for stuff stored on S3.

I would give you more pointers, but I'm myself very unsure about what the best way to handle that case would be.

margori · 2015-02-25T05:16:05Z

New try.

In this opportunity, photo file is gotten through unprocessed photo uploader.

margori · 2015-02-25T05:25:49Z

This PR is related to #5343

jhass · 2015-02-25T10:18:43Z

app/models/user.rb

+        begin
+          zos.put_next_entry(photo.remote_photo_name)
+          zos.print(photo.unprocessed_image.file.read)
+        rescue


Do you have a more specific exception that you expect? Or even a better, a way to guard the code raising to from running in that case?

If file doesn't exists it returns Errno::ENOENT.

margori · 2015-02-26T21:02:49Z

Done: redirecting when download_photos is called.

In this case, filename should be unguessable. Then new request are stored with random filename.

jhass · 2015-02-26T21:27:43Z

app/uploaders/exported_photos.rb

+  def secure_token(length=16)
+    var = :"@#{mounted_as}_secure_token"
+    model.instance_variable_get(var) or model.instance_variable_set(var, SecureRandom.hex(length/2))
+  end


That's just 16**6 combinations, I see no reason to limit it that much.

margori · 2015-02-27T00:07:35Z

Filename token increased to 128bits.

margori · 2015-03-02T05:54:20Z

Re-based and code updated for new gems.

Need review. Thanks!

jhass · 2015-03-02T12:18:50Z

app/models/user.rb

+    Zip::ZipOutputStream.open(temp_zip.path) do |zos|
+      photos.each do |photo|
+        begin
+          photo_data = photo.unprocessed_image.file.read


Looking good so far, I guess we just have to see how it behaves on S3 (+ Heroku) setup. I still don't understand how this call pulls the photo from S3 tbh. If you're able to clarify that... :)

Another last small remark I have is moving the temp_zip.close below to an ensure block.

This is what a thought:

If I delete a post with a photo in it, photo file is deleted from my disc.

I don't found code that delete (unlink) the file. So there's an object that knows what to do.

The uploader field may know where to upload and to delete it.

If so, it may also know where to read it.

I tried it and worked, at least in my local file system.

I found that admin can choose where to store files by setting carrierwave config file. I didn't try over S3. Amazon wants my credit card number for a free account and... no, darling.

What I really want to know is how to create a test to simulate a S3 storage. Do you have a sample code?

No, I don't have an example test for that, I'm actually not sure whether it's testable, I guess one could enable hosting on S3 in the test and then webmock should catch the network call.

Hit me up on IRC if you want credentials to a S3 bucket for your tests.

I think I fixed the S3 access ;)

Yes, you did. :)

Downloads work as expected, but having problem with upload. CarrierWave use same file name of downloaded image for the new uploaded zip. So upload fails saying "You can't upload a jpg. A zip expected" :(

Mmh, it did upload a zip though:

Good. This capture confirms that one export file is maintained.

margori · 2015-03-03T23:57:13Z

New changes

Removed "zip" extension validation in 'exported_photos.rb'.
Ensured "update exporting_photos: false" in case of errors.

These changed were successfully tested in AWS S3.

Export user photos

jhass · 2015-03-04T00:46:09Z

Alright, thank you!

Flaburgan · 2015-03-04T07:46:28Z

Nice, thank you!

margori · 2015-03-04T09:55:02Z

UHU! Thank you too!

goobertron · 2015-03-04T12:17:58Z

Hey, thanks very much, @margori!

margori mentioned this pull request Feb 19, 2015

Remove photo export from settings #5634

Merged

margori force-pushed the 1802_export_photos branch from d251037 to 50dd131 Compare February 20, 2015 19:07

jhass reviewed Feb 20, 2015
View changes

margori force-pushed the 1802_export_photos branch from 50dd131 to 281cafd Compare February 23, 2015 03:36

jhass reviewed Feb 23, 2015
View changes

margori force-pushed the 1802_export_photos branch from 281cafd to e6ee9f8 Compare February 25, 2015 05:05

jhass reviewed Feb 25, 2015
View changes

margori force-pushed the 1802_export_photos branch 5 times, most recently from 31712a2 to 5a36b0f Compare February 26, 2015 11:49

jhass reviewed Feb 26, 2015
View changes

margori force-pushed the 1802_export_photos branch from 5a36b0f to 236103e Compare February 27, 2015 00:04

jhass mentioned this pull request Feb 27, 2015

Update the help section #5698 #5706

Merged

margori force-pushed the 1802_export_photos branch 2 times, most recently from c3105b2 to dda3d3b Compare February 28, 2015 04:31

margori changed the title ~~Exports user photos as zip file~~ Export user photos Feb 28, 2015

margori force-pushed the 1802_export_photos branch 3 times, most recently from 7f63fd2 to 5141c0a Compare March 2, 2015 05:51

jhass reviewed Mar 2, 2015
View changes

Exports user photos as zip file

b154d87

margori force-pushed the 1802_export_photos branch from 5141c0a to b154d87 Compare March 3, 2015 23:50

jhass added this to the next-major milestone Mar 4, 2015

jhass merged commit b154d87 into diaspora:develop Mar 4, 2015

jhass added a commit that referenced this pull request Mar 4, 2015

Merge pull request #5685 from margori/1802_export_photos

abc4203

Export user photos

jaywink mentioned this pull request Mar 15, 2015

User data export / (partial) import metaissue #5343

Open

7 tasks

Export user photos #5685

Export user photos #5685

margori commented Feb 19, 2015

jhass commented Feb 19, 2015

Flaburgan commented Feb 19, 2015

jhass commented Feb 19, 2015

goobertron commented Feb 19, 2015

margori commented Feb 19, 2015

margori commented Feb 19, 2015

margori commented Feb 20, 2015

jhass Feb 20, 2015

margori Feb 20, 2015

jhass Feb 20, 2015

margori Feb 23, 2015

margori commented Feb 23, 2015

jhass Feb 23, 2015

margori commented Feb 25, 2015

margori commented Feb 25, 2015

jhass Feb 25, 2015

margori Feb 25, 2015

margori commented Feb 26, 2015

jhass Feb 26, 2015

margori commented Feb 27, 2015

margori commented Mar 2, 2015

jhass Mar 2, 2015

margori Mar 2, 2015

jhass Mar 2, 2015

jhass Mar 2, 2015

margori Mar 3, 2015

jhass Mar 3, 2015

margori Mar 3, 2015

margori commented Mar 3, 2015

jhass commented Mar 4, 2015

Flaburgan commented Mar 4, 2015

margori commented Mar 4, 2015

goobertron commented Mar 4, 2015

Export user photos #5685

Export user photos #5685

Conversation

margori commented Feb 19, 2015

jhass commented Feb 19, 2015

Flaburgan commented Feb 19, 2015

jhass commented Feb 19, 2015

goobertron commented Feb 19, 2015

margori commented Feb 19, 2015

margori commented Feb 19, 2015

margori commented Feb 20, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

margori commented Feb 23, 2015

Choose a reason for hiding this comment

margori commented Feb 25, 2015

margori commented Feb 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

margori commented Feb 26, 2015

Choose a reason for hiding this comment

margori commented Feb 27, 2015

margori commented Mar 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

margori commented Mar 3, 2015

jhass commented Mar 4, 2015

Flaburgan commented Mar 4, 2015

margori commented Mar 4, 2015

goobertron commented Mar 4, 2015