Feature/json exporter #5354

gdpelican · 2014-10-19T04:03:06Z

A quick first pass at a JSON exporter. Would welcome some feedback around which fields are required, useful, etc.

Note that this hasn't touched the XML export functionality at all.

jaywink · 2014-10-19T08:56:23Z

Awesome, thanks for working on this extremely important feature!

Will check in more detail later.

As a general comment relating to exporters, I guess ideally we should aim to push them to Sidekiq for processing - and then email the user a link to the file (auto-expiry of file). But would this be a problem for pods on Heroku for example? Solved with S3 support?

gdpelican · 2014-10-19T09:15:47Z

Yeah, thought about emailing them their data, but can't do it directly with the potential json size. Heroku would probably preclude storing it locally, but uploading could be a solution.

Maybe after the file's done processing we create an Export model, which uploads the file to S3 and auto-explodes in like 3 days, James Bond-style.

class Export < ActiveModel::Base
  belongs_to :user
  has_attached_file :exported_data
  after_create :schedule_deletion

  def schedule_deletion
    # schedule another sidekiq to destroy this model 
  end
end

??

jhass · 2014-10-19T10:41:41Z

In either case, we should consider running the result through gzip, that'll drastically reduce the filesize.

jaywink · 2014-11-22T12:08:15Z

OK revisiting this, sorry for taking so long.

I think initially it's ok if we build 'n' serve the file as it is done now in runtime as long as we don't have large data included - eg comments and posts. When those arrive, we need to push the processing into sidekiq.

Agreed?

Also, the JSON isn't actual JSON in my env at least, it's like:

{:name=>"jaywink@localhost:3000", :email=>"mail@jasonrobinson.me", :serialized_private_key=>"-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAtMQTCTnPaUCzoNSzDFlZSoRXHEOkHjcBoXybUs7K15Ny9Bbz\nnbvrQmhG7LUxJ29On8jkvnHe7JWMfLztIHMjMe2sqcruo003KcahT2i9I3Zd6nSm\n
.....

And so on.

Otherwise, once the format and what attributes to include have been tweaked - would love to get this merged!

jaywink · 2014-11-22T12:16:44Z

Also one other thing. I think we should include the diaspora* version that exported the file from day one. The tricky thing is how to get this, since any develop version pods would just report head.

@jhass any ideas? Kinda difficult how we replace the version in defaults.version.number currently.. Having the real version number there always would be nice and then having defaults.version.head True/False would be something that would work better - but I guess that would change quite many things..

jaywink · 2014-11-22T12:19:50Z

app/models/user.rb

@@ -45,6 +47,8 @@ class User < ActiveRecord::Base
  has_many :invitations_from_me, :class_name => 'Invitation', :foreign_key => :sender_id
  has_many :invitations_to_me, :class_name => 'Invitation', :foreign_key => :recipient_id
  has_many :aspects, -> { order('order_id ASC') }
+  has_many :posts, through: :person


If we outscope posts and comments from this pull, these two should be removed to be added later.

jhass · 2014-11-22T14:35:48Z

Maybe we should just give the export format its own version. Including the current release won't help much for the dev version, since you still don't know if it was 0.5.0.0dev prior the breaking change or 0.5.0.0dev after the breaking change.

gdpelican · 2014-11-28T11:36:00Z

Alright, finally got some time to sit down with this for a little bit. Should be ready for another look @jaywink

jaywink · 2014-12-07T13:25:30Z

lib/diaspora/exporter.rb

+    def strategy(format)
+      @strategy ||= case format
+        when :json then Exporters::JSON
+        when :xml  then Exporters::XML


Can we just drop everything related to the old XML user exporter? A community decision already accepted XML as the new format, the old one doesn't really work anyway.

Objections anyone?

A community decision already accepted XML as the new format

Shurely JSON, not XML?

Yeah :D

On 7 December 2014 16:19:03 EET, goob notifications@github.com wrote:

@@ -5,12 +5,24 @@
module Diaspora

class Exporter

def initialize(strategy)

self.class.send(:include, strategy)

def initialize(format = :json)

self.class.include strategy(format) if
strategy(format).present?

end

def strategy(format)

@strategy ||= case format

when :json then Exporters::JSON

when :xml then Exporters::XML

A community decision already accepted XML as the new format

Shurely JSON, not XML?

Reply to this email directly or view it on GitHub:
https://github.com/diaspora/diaspora/pull/5354/files#r21424879

Br,
Jason Robinson
https://jasonrobinson.me

jaywink · 2014-12-07T13:37:56Z

Thanks @gdpelican I'll check the export asap on my dev environment, hopefully tonight. I think this is pretty close, personally I think we should drop the XML stuff already. If no one objects within a few days, feel free to do it IMHO :)

jaywink · 2014-12-07T21:28:28Z

Quick test in dev env, looks ok to me. Note it's not prettyfied of course normally, I prettified it.

One thing that needs fixing is that contacts don't show which aspect they have been added to. The one contact here below is in the "Friends" aspect, but the json doesn't show that in any way.

{
    "name": "foobar@localhost:3000",
    "email": "foobar@example.com",
    "language": "en",
    "username": "foobar",
    "disable_mail": false,
    "show_community_spotlight_in_stream": true,
    "auto_follow_back": true,
    "auto_follow_back_aspect": "Acquaintances",
    "profile": {
        "first_name": "",
        "last_name": "",
        "gender": "",
        "bio": "### Markdown header\r\n\r\n> quote\r\n\r\n   code para\r\n\r\nFoo",
        "location": "Hki",
        "image_url": "/assets/user/default.png",
        "diaspora_handle": "foobar@localhost:3000",
        "searchable": true,
        "nsfw": false
    },
    "aspects": [{
        "name": "Family",
        "contacts_visible": true,
        "chat_enabled": false
    }, {
        "name": "Friends",
        "contacts_visible": true,
        "chat_enabled": false
    }, {
        "name": "Work",
        "contacts_visible": true,
        "chat_enabled": false
    }, {
        "name": "Acquaintances",
        "contacts_visible": true,
        "chat_enabled": false
    }],
    "contacts": [{
        "sharing": false,
        "receiving": true,
        "person_guid": "bed18bf02ee001323b1010c37b6c3c6f",
        "person_name": "jaywink@localhost:3000",
        "person_first_name": "jaywink",
        "person_diaspora_handle": "jaywink@localhost:3000"
    }]
}

jaywink · 2014-12-07T21:29:14Z

Also, date of birth is missing from the profile fields :)

jaywink · 2014-12-07T21:40:37Z

Also, something odd related to the exporter, but that is probably the fault of the profile system and not the exporter itself. Just after creating the profile it looks like this:

    "first_name": null,
    "last_name": null,

But after saving it once, it looks like this:

    "first_name": "",
    "last_name": "",

So we probably have a bug in profile creation, setting fields to null, since the save does things differently.

gdpelican · 2014-12-08T09:11:17Z

@jaywink

Cool, so I

Added version 1.0 to the root of the exported json, pushing everything else into 'user'
Added birthday to the profile serializer
Added aspects to the contact serializer
Nuked the XML serializer entirely

Didn't get a chance to look at the null / "" profile name beyond having a chuckle at the checkbox tags wtf comment in the profiles_controller. (checkbox tags are, indeed, wtf.)
Strikes me as pretty low on the scale of broken?

Flaburgan · 2014-12-08T09:34:00Z

Yeah I don't think you should worry about that in this PR.

jaywink · 2014-12-08T21:08:39Z

app/models/user.rb

@@ -37,6 +37,8 @@ class User < ActiveRecord::Base
  serialize :hidden_shareables, Hash

  has_one :person, :foreign_key => :owner_id
+  has_one :profile, through: :person


Quite a few tests are failing. I'm wondering if this is the cause? For example: AccountDeleter has all user association keys accounted for

Diff: @@ -13,7 +13,6 @@ :invited_by, :notifications, :person, - :profile, :reports, :services, :tag_followings, # ./spec/lib/account_deleter_spec.rb:175:in `block (2 levels) in <top (required)>'

So the tests need a little check through, whether things are really "breaking" or just tests need patching.
Also, if you could clean the commit message and add a Changelog entry - I think this would be ready for merging.

gdpelican · 2014-12-11T07:55:40Z

Pushed some test fixes, it seems like ActiveModelSerializer changes the default behaviour of render :json => @model a little bit; I was able to fix the failing tests by adding root: false to them.

(difference being {id: 1, name: 'name}with root: false, and {user: {id: 1, name: 'name'}}' with the defaulted root: true

Wonder if we should just turn off using the root element across the board for AMS, or if the tests passing is confidence enough that all the important json is still coming through correctly?

@jaywink

gdpelican · 2014-12-11T08:03:19Z

Seems like there's still some frontend tests to fix up; I'll fix em up when I get the chance.

gdpelican · 2014-12-11T08:23:16Z

Cool, looks like that build's passing now.

Squashed 'er up and put in a message in the changelog as well.

Flaburgan · 2014-12-11T08:38:41Z

Hey, merge is close to you... :D

jhass · 2014-12-11T12:05:18Z

Changelog.md

+## Json exporter functionality (Iteration 1)
+
+Allows users to export their data in JSON format from their user settings page
+


It's an important feature, but the big headings are for things the podmin needs to read and know for update ;)

jhass · 2014-12-11T12:11:42Z

Just some minor remarks, then this should be good :)

jhass · 2014-12-15T15:53:38Z

spec/lib/account_deleter_spec.rb

+        @profile_deletion.perform!
+        expect(@profile.reload.first_name).to be_blank
+        expect(@profile.reload.last_name).to be_blank
+        expect(@profile.reload.searchable).to be_falsey


Actually, slight optimization nitpick here, reload returns self and updates the receiver, so just do it one time above the expectations.

gdpelican · 2014-12-16T07:19:25Z

Ready for final sign-off I think @jaywink @jhass

jhass · 2014-12-17T15:45:15Z

Alright, thank you!

Feature/json exporter

goobertron · 2014-12-17T17:40:25Z

Thanks for your work on this, @gdpelican (and @jhass too).

Flaburgan · 2014-12-17T18:02:41Z

Nice! The first step for #5343 :D

Thank you @gdpelican @jhass and @jaywink

Flaburgan · 2014-12-17T18:24:04Z

Just pulled this on d-fr, I tested it with my account (more than 1000 contacts). I put it in http://jsonlint.com/ and it says "valid" :)

svbergerem · 2014-12-17T21:04:46Z

Awesome! Nice work @gdpelican!

jaywink reviewed Nov 22, 2014
View changes

jaywink mentioned this pull request Nov 22, 2014

User data export / (partial) import metaissue #5343

Open

7 tasks

jaywink reviewed Dec 7, 2014
View changes

gdpelican force-pushed the feature/json-exporter branch 3 times, most recently from f0035fe to e976fae Compare December 8, 2014 09:02

jaywink reviewed Dec 8, 2014
View changes

gdpelican force-pushed the feature/json-exporter branch from 68207df to bd8e524 Compare December 11, 2014 08:22

jhass reviewed Dec 11, 2014
View changes

gdpelican force-pushed the feature/json-exporter branch 2 times, most recently from d417fb0 to 4ba9a64 Compare December 15, 2014 12:07

jhass reviewed Dec 15, 2014
View changes

Add JSON exporter for user profile download

e174514

gdpelican force-pushed the feature/json-exporter branch from 4ba9a64 to e174514 Compare December 15, 2014 19:56

jhass added this to the next-major milestone Dec 17, 2014

jhass closed this Dec 17, 2014

jhass reopened this Dec 17, 2014

jhass added a commit that referenced this pull request Dec 17, 2014

Merge pull request #5354 from gdpelican/feature/json-exporter

6806b2d

Feature/json exporter

jhass merged commit 6806b2d into diaspora:develop Dec 17, 2014

jaywink mentioned this pull request Jan 10, 2015

Export function produces malformed XML #4032

Closed

ghost mentioned this pull request Jan 24, 2015

Post date to json export #5585

Closed

Feature/json exporter #5354

Feature/json exporter #5354

gdpelican commented Oct 19, 2014

jaywink commented Oct 19, 2014

gdpelican commented Oct 19, 2014

jhass commented Oct 19, 2014

jaywink commented Nov 22, 2014

jaywink commented Nov 22, 2014

jaywink Nov 22, 2014

jhass commented Nov 22, 2014

gdpelican commented Nov 28, 2014

jaywink Dec 7, 2014

goobertron Dec 7, 2014

jaywink Dec 7, 2014

jaywink commented Dec 7, 2014

jaywink commented Dec 7, 2014

jaywink commented Dec 7, 2014

jaywink commented Dec 7, 2014

gdpelican commented Dec 8, 2014

Flaburgan commented Dec 8, 2014

jaywink Dec 8, 2014

gdpelican commented Dec 11, 2014

gdpelican commented Dec 11, 2014

gdpelican commented Dec 11, 2014

Flaburgan commented Dec 11, 2014

jhass Dec 11, 2014

jhass commented Dec 11, 2014

jhass Dec 15, 2014

gdpelican commented Dec 16, 2014

jhass commented Dec 17, 2014

goobertron commented Dec 17, 2014

Flaburgan commented Dec 17, 2014

Flaburgan commented Dec 17, 2014

svbergerem commented Dec 17, 2014

		## Json exporter functionality (Iteration 1)

		Allows users to export their data in JSON format from their user settings page

Feature/json exporter #5354

Feature/json exporter #5354

Conversation

gdpelican commented Oct 19, 2014

jaywink commented Oct 19, 2014

gdpelican commented Oct 19, 2014

jhass commented Oct 19, 2014

jaywink commented Nov 22, 2014

jaywink commented Nov 22, 2014

jaywink Nov 22, 2014

Choose a reason for hiding this comment

jhass commented Nov 22, 2014

gdpelican commented Nov 28, 2014

jaywink Dec 7, 2014

Choose a reason for hiding this comment

goobertron Dec 7, 2014

Choose a reason for hiding this comment

jaywink Dec 7, 2014

Choose a reason for hiding this comment

jaywink commented Dec 7, 2014

jaywink commented Dec 7, 2014

jaywink commented Dec 7, 2014

jaywink commented Dec 7, 2014

gdpelican commented Dec 8, 2014

Flaburgan commented Dec 8, 2014

jaywink Dec 8, 2014

Choose a reason for hiding this comment

gdpelican commented Dec 11, 2014

gdpelican commented Dec 11, 2014

gdpelican commented Dec 11, 2014

Flaburgan commented Dec 11, 2014

jhass Dec 11, 2014

Choose a reason for hiding this comment

jhass commented Dec 11, 2014

jhass Dec 15, 2014

Choose a reason for hiding this comment

gdpelican commented Dec 16, 2014

jhass commented Dec 17, 2014

goobertron commented Dec 17, 2014

Flaburgan commented Dec 17, 2014

Flaburgan commented Dec 17, 2014

svbergerem commented Dec 17, 2014