User data export / (partial) import metaissue

[jaywink]

So to possibly get things moving, I'm creating here an issue for everything needed for user data export / (partial) import to happen. We've got good experience with splitting large items into independant tasks in issues like this - so hopefully this will help things forward and some day there will be a working export / (partial) import feature. I say partial because until the migration thingy is reality, it's not really feasible to import posts - but profile data should be no issue.

Note! Please contribute general discussion relating to this on Loomio here: https://www.loomio.org/d/w6nfQPFc/personal-data-import-export

Please feel free to work on single items and suggest smaller tasks in addition to these if necessary.

User data - export

• Add user data export in JSON format (proposal, proposal) Feature/json exporter #5354
• Add contacts, aspects and followed tags to data export Feature/json exporter #5354
• Remove old XML export

User data - import

• Import for user data (profile, settings, etc, as exported)
• Import for contacts (automatic creation of aspects and adding contacts)

User content - export

• Add posts and comments export to the user data export (pull JSON Exporter, part deux #5499 )
• Add/fix photos export tool (pull Export user photos #5685)

[Faldrian]

It might be a different issue... but user migration from one pod to the other would still be a much needed feature.
If I do this by exporting and importing a user, I have a a user account with contacts but none of my previous posts. Maybe it would be good to have some migration via federation, so the origin pod can issue a "user migrated to newusername@newpodaddress.tld" command and every pod with federated posts can rewrite that user information to match the new location. This way the former structure keeps intact. :)
(This needs an existing target-user-account with some kind of auth-codes between the two pods involved in the migration... maybe inspired by internet domain transfer)

(I will remove my comment here if there is a place where it's better suited... I think there is, but I have no overview on where which discussion regarding this topic is happening.)

[jaywink]

@Faldrian you're thinking of #908 .

But slightly related - if account backup and restore ie migration was a real thing - do we really need an import for the JSON exports? The exports are needed as they are anyway, but importing them kinda becomes irrelevant imho.

[Faldrian]

@jaywink I have no clue what this was / is all about... it's a year since my comment above, I have forgotten everything and have no intention to re-read the topic... so you have a plan, do stuff, make us happy :)

[jaywink]

The latest automatic back up / restore spec proposal includes the import parts in this issue. Would be fancy to get more comments on it, split it into smaller chunks and dump a lot of bounty money on the issues - maybe get something to happen :)

Anyone wanting to give their insight, please check the proposal and comment in issue #908 . If no comments are received after some time I could just create a proposal to accept the proposal as a guideline for development.

[tclaus]

With #8274 this might be the missing "Seed wizard" ?

[dredmorbius]

Include canonical post URL in JSON data exports.

An issue I've repeatedly encountered with Diaspora* data export is that the provided JSON format does not provide either the short or the canonical URL for the post in question, nor so far as I can tell, any means for determining what that URL might be / have been.

Whilst the export does provide the content, what it does not provide is a means to supply a list of URLs to an archival service such that a given profile might be archived elsewhere online, the prime examples being at the Internet Archive Wayback Machine (IAWBM) via its "save URL now" URL (https://web.archive.org/save/<URL>).

The ability to do just that was critically useful during the shutdown of Google+, which otherwise offered a similar data export mechanism. It's proving a critical stumbling point with the recently announced planned shutdown of one of the oldest and largest Diaspora* instances, Joindiaspora.com, of which I've been a 10-year-plus active member.

REQUEST:: Add post URL to the JSON data export, preferably the long-form version of that URL.

E.g.,

• DO include the long form:
  https://joindiaspora.com/posts/b4fae69034500139f93f002590d8e506
• Do NOT include the short form: https://joindiaspora.com/posts/19830343

[Flaburgan]

Hello @dredmorbius, the JSON export contains the GUID of every element, if you want to rebuild the link, you just have to add https://joindiaspora.com/posts before the post GUID.

[dredmorbius]

Thanks, I'll look into that. I need to wrestle with the JSON and jq parsing. (I've done that before, it's always a bit of an adventure.)

…

-- Edward Morbius ***@***.***) Dr. Edward Morbius's Lair of the Id https://dredmorbius.reddit.com Sent from ProtonMail mobile

-------- Original Message --------

On Jan 11, 2022, 02:07, Fla wrote: Hello ***@***.***(https://github.com/dredmorbius), the JSON export contains the GUID of every element, if you want to rebuild the link, you just have to add https://joindiaspora.com/posts before the post GUID. — Reply to this email directly, [view it on GitHub](#5343 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABJRRYU2JECZELB3EIRUI2DUVPQL5ANCNFSM4AV2F3DA). Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you were mentioned.Message ID: ***@***.***>

[FiXato]

Thanks, I'll look into that. I need to wrestle with the JSON and jq parsing. (I've done that before, it's always a bit of an adventure.)

This one-liner should give you the combined post URLs for every one of your posts from your username_data.json archive:
jq -r '.user .posts[] .entity_data | "https://REPLACE_WITH_YOUR_INSTANCE_DOMAIN/posts/\(.guid )"' username_data.json

If you want to include the created_at and text of each post:
jq '.user .posts[] .entity_data | "https://REPLACE_WITH_YOUR_INSTANCE_DOMAIN/posts/\(.guid) \(.created_at) \(.text)"' username_data.json