JSON Exporter, part deux #5499

gdpelican · 2014-12-28T03:10:47Z

Thought I'd share this for a little feedback if you'd care to give it a look @jhass @jaywink

This currently:

Boots the current export off to Sidekiq
Uploads a json file through carrierwave
Sends an email notifying the user when the job's complete, which includes a link to the upload
Should pass tests, although we'll see when Travis gets its mitts on it.

It doesn't:

Have a good UI solution yet (you just click on 'download my profile' and it takes you to a blank screen with status 200; we'll have to do a flash notice or something to tell the user what's happening)
Include any of the heavy lifting associations we outscoped from the last PR

gdpelican · 2014-12-28T03:19:30Z

app/models/user.rb

@@ -505,6 +518,6 @@ def clearable_fields
                            "created_at", "updated_at", "locked_at",
                            "serialized_private_key", "getting_started",
                            "disable_mail", "show_community_spotlight_in_stream",
-                            "email", "remove_after"]
+                            "email", "remove_after", "export"]


Should export get cleared when we do user.clear_account!? I'd imagine so...

Sounds right.

jhass · 2014-12-28T17:16:57Z

Speaking of the mails, we should keep in mind that sending mails is entirely optional and not configured by default.

gdpelican · 2014-12-29T00:52:34Z

Hm, I don't think sending emails should be a hard dependency for this feature, it's more of a nice-to-have.

How about something like this:

When you visit your profile page

If you have not requested an export
There is a link to request an export

If you have requested an export and it is not complete
There is a message to the effect of 'We're still working on your export, please check back soon'

If you have requested an export and it is complete
There is a link to download your export
And a link which says 'Update my export', with a timestamp of the last created export

Flaburgan · 2014-12-29T08:38:56Z

That looks like a nice solution ;)

gdpelican · 2014-12-29T12:41:34Z

So I've added

Sending out a failure email for if export fails to upload for some reason
A frontend workflow which looks like this:

User who hasn't requested an export:

User who has an export pending:

User who has requested a completed export:

Next up would be throwing in the big json chunks (posts and comments) to the serializer, and having a look at gzipping the file we generate.

gdpelican · 2015-01-06T17:05:49Z

Alright, I've changed it so the data is GZipped and added in posts and comments to the serializers (might need some feedback on which fields want to be there)

It'd be cool to get your take on this, @jaywink ?

jaywink · 2015-01-10T14:04:07Z

app/models/user.rb

+  end
+
+  def perform_export!
+    export = Tempfile.new([username, '.json'], encoding: 'ascii-8bit')


Hmm why is the encoding ascii-8bit?

ASCII-8bit is Rubys clunky way of saying BINARY (which is an alias for ASCII-8bit in fact). Remember, ASCII is a 7bit encoding ;) Since we write gzipped data here, that's correct.

jaywink · 2015-01-10T14:25:42Z

Other than the comments relating to the attributes, super awesome work 🌟 A lot of people will be happy to have this working again, as it's a shame our export is currently worse than many proprietary networks ;)

🍶 for you ;)

jaywink · 2015-01-10T14:31:28Z

Oh also, I'm assuming this will work fine with pods using S3 for storage etc?

jhass · 2015-01-10T14:38:47Z

app/serializers/export/post_serializer.rb

+               :image_width,
+               :likes_count,
+               :comments_count,
+               :reshares_count,


Counter caches, shouldn't be included.

Doesn't it contain the current (when exported) count though? IMHO that is relevant information for the user :)

It should, I don't see much value in that information though.

I think it could be valuable for some - imagine you want to look at some old export of your data from an old pod for example. Maybe someone will make an app to analyse exports and rank posts with most likes etc.

jaywink · 2015-01-10T17:49:42Z

Also, before merging, the changelog entry for the export should probably be refreshed. I guess the export version does not need to be bumped since the current export version hasn't gone stable yet.

gdpelican · 2015-01-10T23:25:53Z

I haven't been able to test using an S3 instance, but I just aped the Carrierwave uploaders for images, so it shouldn't behave any differently.

gdpelican · 2015-01-12T03:48:00Z

Well, I b0rked the rebase a little, but merged in develop, updated the serializers & changelog, and changed that one symbol that wasn't supposed to be there.

Also made it so we export files in a .json.gz file, which 7zip should handle.

jaywink · 2015-01-16T22:51:36Z

Alright, no more comments to this and looks awesome to me - merging! Nice to get this in to the next release, thanks @gdpelican !

JSON Exporter, part deux

Flaburgan · 2015-01-16T22:56:04Z

Awesome!

Flaburgan · 2015-01-16T23:40:40Z

Currently in test on diaspora-fr :)

marienfressinaud · 2015-01-17T10:22:10Z

@Flaburgan > I tried yesterday night but it still shows me "We are currently processing your data. Please check back in a few moments." this morning. I'm pretty sure there is a problem ;).

gdpelican · 2015-01-17T10:37:51Z

Oh noes! I'll have a poke at it tomorrow to see what's up. We're sure
sidekiq is running?

On Sat, Jan 17, 2015 at 11:22 PM, Marien Fressinaud <
notifications@github.com> wrote:

@Flaburgan https://github.com/Flaburgan > I tried yesterday night but
it still shows me "We are currently processing your data. Please check back
in a few moments." this morning. I'm pretty sure there is a problem ;).

—
Reply to this email directly or view it on GitHub
#5499 (comment).

Flaburgan · 2015-01-17T10:57:59Z

Yeah sidekiq is running, Marien and my jobs are listed in "Enqueued". They are the only two jobs in the queue.

jhass · 2015-01-17T12:14:07Z

export_user wasn't added to https://github.com/diaspora/diaspora/blob/develop/config/sidekiq.yml#L6

Flaburgan · 2015-01-17T12:38:41Z

Confirmed, adding export_user there immediately made the jobs processed. But the message on the panel is still We are currently processing your data. Please check back in a few moments.

marienfressinaud · 2015-01-17T12:40:40Z

Indeed but I have a different message: download my profile (Last updated at 2015-01-17 12:36:04 UTC) refresh my profile data so it seems ok for me. Nevertheless there are other bugs, should we open new specific tickets for each bug found?

jhass · 2015-01-17T12:49:55Z

should we open new specific tickets for each bug found?

Yes, sure.

jaywink · 2015-01-17T18:57:00Z

@gdpelican the sidekiq queue thingy is fixed now, no need to check that :)

gdpelican reviewed Dec 28, 2014
View changes

jaywink reviewed Jan 10, 2015
View changes

jhass reviewed Jan 10, 2015
View changes

jaywink mentioned this pull request Jan 10, 2015

User data export / (partial) import metaissue #5343

Open

7 tasks

gdpelican force-pushed the feature/json-exporter-deux branch from e24431f to 289ca3a Compare January 12, 2015 02:37

Add contacts/posts, and GZipping JSON exporter output

1c69dd7

gdpelican force-pushed the feature/json-exporter-deux branch from 4286062 to 1c69dd7 Compare January 14, 2015 11:20

jaywink added a commit that referenced this pull request Jan 16, 2015

Merge pull request #5499 from gdpelican/feature/json-exporter-deux

6513053

JSON Exporter, part deux

jaywink merged commit 6513053 into diaspora:develop Jan 16, 2015

jaywink added this to the next-major milestone Jan 16, 2015

This was referenced Jan 17, 2015

[JSON exporter] bad email formatting #5567

Closed

[JSON exporter] Last updated information is overflowing #5568

Closed

jaywink mentioned this pull request Jan 17, 2015

Add missing export_user queue to sidekiq config #5570

Merged

Flaburgan mentioned this pull request Feb 19, 2015

Export user photos #5685

Merged

JSON Exporter, part deux #5499

JSON Exporter, part deux #5499

gdpelican commented Dec 28, 2014

gdpelican Dec 28, 2014

jhass Dec 28, 2014

jhass commented Dec 28, 2014

gdpelican commented Dec 29, 2014

Flaburgan commented Dec 29, 2014

gdpelican commented Dec 29, 2014

gdpelican commented Jan 6, 2015

jaywink Jan 10, 2015

jhass Jan 10, 2015

jaywink commented Jan 10, 2015

jaywink commented Jan 10, 2015

jhass Jan 10, 2015

jaywink Jan 10, 2015

jhass Jan 10, 2015

jaywink Jan 10, 2015

jaywink commented Jan 10, 2015

gdpelican commented Jan 10, 2015

gdpelican commented Jan 12, 2015

jaywink commented Jan 16, 2015

Flaburgan commented Jan 16, 2015

Flaburgan commented Jan 16, 2015

marienfressinaud commented Jan 17, 2015

gdpelican commented Jan 17, 2015

Flaburgan commented Jan 17, 2015

jhass commented Jan 17, 2015

Flaburgan commented Jan 17, 2015

marienfressinaud commented Jan 17, 2015

jhass commented Jan 17, 2015

jaywink commented Jan 17, 2015

JSON Exporter, part deux #5499

JSON Exporter, part deux #5499

Conversation

gdpelican commented Dec 28, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhass commented Dec 28, 2014

gdpelican commented Dec 29, 2014

Flaburgan commented Dec 29, 2014

gdpelican commented Dec 29, 2014

gdpelican commented Jan 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaywink commented Jan 10, 2015

jaywink commented Jan 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaywink commented Jan 10, 2015

gdpelican commented Jan 10, 2015

gdpelican commented Jan 12, 2015

jaywink commented Jan 16, 2015

Flaburgan commented Jan 16, 2015

Flaburgan commented Jan 16, 2015

marienfressinaud commented Jan 17, 2015

gdpelican commented Jan 17, 2015

Flaburgan commented Jan 17, 2015

jhass commented Jan 17, 2015

Flaburgan commented Jan 17, 2015

marienfressinaud commented Jan 17, 2015

jhass commented Jan 17, 2015

jaywink commented Jan 17, 2015