Seed Migration Wizard #908

Open
diaspora-redmine-github-migration opened this Issue Jul 8, 2011 · 86 comments

Projects

None yet
@diaspora-redmine-github-migration

Issue 505 from bugs.joindiaspora.com
Created by: Wavesonics
On Mon Nov 1 20:03:57 2010

Priority: Normal
Status: New

There should be a wizard in the Account UI for migrating your seed from one pod to the another, all server to server.

This way contacts could be notified of the move in a secure way, and the old account could be deleted when the move was complete.

There is a $200 open bounty on this issue. Add to the bounty at Bountysource.

@diaspora-redmine-github-migration

Comment by: Hexagon
On Mon Nov 1 20:45:46 2010

+1

@diaspora-redmine-github-migration

Comment by: Cathryne
On Fri Nov 26 18:43:11 2010

+1 with a user-defined overlap time until the old account automatically closes.

@luzvioleta

This is sincerely critical .... any idea on the state of this ? (or the partial photo & xml export ? )

@Nygu
Nygu commented Aug 1, 2011

It's not being worked on yet. I guess it'll come somewhere during beta. On the one hand I think it's a shame, on the other hand I'm glad they are working on more usable things first.

@ghost
ghost commented Sep 1, 2011

+1

@pniedzielski

I'm willing to work on this with someone else. Have a good amount of programming experience, but just learned Ruby...also, due to classes, unable to work on it every day.

Anyone else willing to help with this?

@sebastianbuettrich

If there s need for it, i d be willing to help with this:

relevant experience:
databases, replication, migration (12 yrs mysql, 15 yrs general sql), db architecture,
coding: C, perl, php

no skills in ruby at all :(

@pniedzielski

I've sent ideas to diaspora-dev in this mail thread, thanks for the discussion, guys!

@sebastianbuettrich

great. i ll gladly contribute where it s needed -
i m blank on ruby though.

my feeling is that we ll have to look a little bit at database structure, redundancies, in the process -
though you rightfully say it s out of scope here -

but it s something i ll gladly comment on / help with.

@Wavesonics

Any progress on this? Seems like an important feature before going to beta.

@pniedzielski

It's not on the beta checklist, and though it is an important feature, I don't think it need be -- Diaspora's built on an iterative development process, so "Beta" is much more fluid (and possibly having less substantial meaning) here than it is in other sorts of Free Software projects. That last sentence is my opinion only, so it's fine if you feel differently.

More directly, I'm trying to learn the internals of Diaspora (strong documentation is...lacking), before I attempt this, as well as coordinating a release for my own Free Software Project in the next two weeks. I plan to attack it around Thanksgiving (US one, three weeks away). That's my approximate timeframe.

@stephens2424

I'm not sure if or when I'd be able to help out on this, but whoever works on this, it would be ideal if the system could take care of updating all your followers/friends of the move for you. Transparently, too, so the remote pods will just receive an update when they next check for information. It just seems important that this doesn't recreate the hassle of changing email addresses or phone numbers.

@zeorin
zeorin commented Jan 1, 2012

Perhaps it would be a good idea if there is functionality to not only move accounts, but in the same process to merge accounts...
And perhaps to sync more than one instance of an account between two or more servers: for redundancy purposes.

@twoolie
twoolie commented Jan 3, 2012

Would the move generate a HTTP 301 Permanent Redirect to the new page location, or would the old pod mirror the updates from the new pod?

@zeorin
zeorin commented Jan 5, 2012

That would depend on whether one moved or merged the account (301) or synced them (mirror). Although you should be able to log into either pod if you're syncing.

@jhass
Member
jhass commented Jan 20, 2012

Follow up #2616

@rriemann

Hey, is this really abandoned? You advertised this feature on your very fist landing page:
http://diasporaproject.org/

@sysfu
sysfu commented Apr 26, 2012

I would like to see this feature implemented soon

@stragu
Contributor
stragu commented Jun 3, 2012

Yes, this would be great. What would happen if a pod closes down or doesn't update the version of Diaspora??

@darklajid

This issue is closed but linked to from

http://diasporaproject.org/#benefits

#2616 (mentioned as 'follow-up') is closed as well, so the only 'status' might potentially exist in a mailing list archive.

Wouldn't it make sense to update the home page to stop sending people here? I'd vote to remove that 'coming soon' feature, because it seems it's not. At the very least the link to this specific issue should be removed.

@creatinglake

This is essential for the long-term viability of the Diaspora mission; the whole point ti to have freedom of movement. Still, great job Diaspora HQ.

@omensinger

+1

@rrohrer
rrohrer commented Aug 29, 2012

This feature would be handy. Also since this is clearly dead, the "coming soon" should be removed from diasporaprojects home page.

@Raven24
Member
Raven24 commented Aug 29, 2012

https://groups.google.com/d/topic/diaspora-discuss/zOpARcOPfoM/discussion is the accumulated feature discussion for this issue.
This is still high priority, but apart from many ideas we never really got any code for this...

@JosephBeck

So we can't move pods easily yet? We just have to delete our old account and make a new one? I thought this was going to be a main feature but seems its not completed. I see that I can export data but not import. Least on Diasp.org which I plan to move from soon.

@kqr
kqr commented Oct 15, 2012

I've been waiting quite a while (years, by now) to sign up on a pod since it's unsure if I'll be able to move to a self-hosted pod later on. I was happy when I saw the "Coming soon" on the main page but apparently this is not something that's in the works. Is there any word on what the problem is? The UI? The code? The security?

@jaywink
Contributor
jaywink commented Oct 15, 2012

Even a stupid migrate process would be cool to start off with - just to be able to import old posts (not trigger any federation for them) and do an automatic aspects creation and contact adding.

@ghost
ghost commented Nov 5, 2012

It would be great to have this function! I myself for example would be very interested in migrating from the joindiaspora-pod to a pod here in Europe. I feel that a function like this would make it more convenient for a user to continue using diaspora when their pod may be shutting down. It would be simple and great just to migrate to another pod and keep on interacting like nothing happened. It is a decentralized idea going well together with the idea of the concept of Diaspora*.

@Gimeit
Gimeit commented Nov 15, 2012

The lack of this feature is the only thing keeping me from signing up for and donating to Diaspora. I'm sad to see that its development has apparently been abandoned - seems like a pretty core concept.

@jaywink
Contributor
jaywink commented Nov 15, 2012

I wouldn't say it has been abandoned - it is a constant discussion topic among not only users talking about features but also many devs that write code for Diaspora_. You have to remember that no one writes code for Diaspora_ for a living any more - it's all volunteers. So this kind of big features will take time. It is a must feature that has to be done at some point though, sooner the better.

@rivendale2010
Contributor

some facility for removal of dead handles would be good to include, maybe generate a webfinger on 404 to a guid or whatever :)

@JosephBeck

It is something as simple as an import feature for the data it lets us export, it isn't rocket science. Seed migration was one of their MAIN advertising points and it has yet to be implemented or even trial implemented. Diaspora* just isn't functional enough and uses very unpopular software among developers. It isn't worth many people's hassle. It did what it set out to do, proved you can have decentralized social networks. Problem is, it just doesn't have really any use. Even if they finished it not enough people even know about Diaspora* to matter. Tech. is an industry where you have to stay focused and working, if you fall too far behind you'll be left there and that is what has happened to Diaspora*. For me personally, it just isn't worth the time to set up and work on. Social Networks of all kinds are declining anyways as people lose faith in Facebook, have no use for Twitter, Google+ is only used by google fanboys and their friends, and things like this were never finished with the basic essentials.

@jaywink
Contributor
jaywink commented Nov 15, 2012

Please keep the discussing issues on Github - this is not a discussion forum but a place to follow up on issues relating to Diaspora*. There is no point fretting about the missing feature - it will be implemented when some developer writes the code. All the devs do this on their spare time and without any compensation. If you really demand something to be implement it - do it or find someone to do it! That is the true power of open source :)

@JosephBeck

You seemed to have missed the point, it isn't just a missing feature. It is a flagship feature that has been used in advertising Diaspora* since the start. That is a major issue and should be given utmost attention and it isn't. YOU were discussing what someone else had said and making excuses, and I was doing the same by commenting on what you said and removing the excuses... You can't blame the fact it is volunteer, several volunteer software developers and even OS developers have brand new features working and testing on a daily basis. You can't state something as a flagship feature then ignore it.

@Gimeit
Gimeit commented Nov 16, 2012

JosephBeck hit the nail on the head - from an end-user perspective, the ability to control your own data is THE distinguishing feature of Diaspora over other social networks. Without the option to migrate pods this feature doesn't exist. There is obviously a lot of other work being done on the project, but it's doomed to obscurity as long as the core functionality is crippled.

@jaywink
Contributor
jaywink commented Nov 16, 2012

@JosephBeck @Gimeit no one is ignoring this feature, I don't know a single person who is somehow participating who is ignoring this or doesn't want it done :)

The reason this isn't implemented is not because it is not wanted but because there are a limited amount of devs working on the code and to be frank there are a lot of things to do and fix - this is only one of them. Recently the most active contributors have been concentrating on code cleanup since the whole community governance thing. But the truth is that more devs are needed to keep things rolling - so if you know some Ruby just jump in and help create an awesome open source social network :)

@jaywink
Contributor
jaywink commented Nov 16, 2012

@DeadSuperHero just noticed this issue is closed for some reason - can you open it? :D

@Raven24 Raven24 reopened this Nov 16, 2012
@dskvr
dskvr commented Jan 10, 2013

Without this feature, this project is nearly worthless. I join Pod A, Pod A starts advertising, and changes their privacy policy. I no longer own my data. I delete Pod A and start with Pod B .... Seems pointless. No different than moving from Instagram to something else. Without this feature, Diaspora isn't Diaspora. This topic is 2 years old. Maybe another Kickstarter to get this feature done?

@tubbo
Contributor
tubbo commented Jan 10, 2013

lol I love how every feature on this issue tracker is "urgent" and the project is totally "worthless" without it.

edit: Incidentally, running your own pod is still the easiest way to be a part of DIASPORA. You can own your data, but at this time you must be self-reliant to do so. Soon, we hope to incorporate this feature, but we want to make sure it's done in a safe and robust way.

@dskvr
dskvr commented Jan 10, 2013

I understand that, but it seems that this feature in particular negates the very core concept of Diaspora ... not everybody can run a pod, to assume so would be a serious flaw, and is nearly an admission of defeat. I am stuck on a pod with advertising, a shifting privacy policy and an old version of Diaspora. I cannot switch. If I was one of the 98% of the people in the world who don't do web development and have no empathy or understanding for what it takes, I would simply leave, and NEVER come back. I respect and admire everybody who contributes to this, and am grateful for any and all progress, but there has apparently been minimal progress on this incredibly important front.

@tubbo
Contributor
tubbo commented Jan 10, 2013

@dskvr The only bottleneck in getting a pod up and running at this time is command-line and Rails experience. We're working on an app right now (might even be out there in beta) that will provision new pods on Heroku, so you basically click a button, enter in your Heroku account and API key, and the app does the rest of the work. The other leg of that is what I'm working on, an app that will tag the latest releases on Github and send an API call that redeploys your app. This way, we can actually institute a kind-of continuous integration system, making sure each of our pod servers are up to date all the time. In situations like when the Rails XML/mass-assignment vulnerabilities were exposed this week, it would be super cool if we could ensure security across our ENTIRE network, without needing root access to each box.

That being said, this is still an issue that needs dealing with. I don't think a Github Issue is the place to discuss how to best move forward on this, so I made a Loomio discussion: https://www.loomio.org/discussions/1653

If you're on our Loomio, please vote on the current proposal and if we get a majority amount of "No's", we'll discuss how to best implement this feature.

@jaywink
Contributor
jaywink commented Jan 10, 2013

I really don't see any point in voting about this feature :) Who wouldn't want pod migration to happen?

The problem is no one has done it yet and all the developers are now doing this as a hobby. So until someone picks this up nothing much to debate (except technical ideas of course) :P

@tubbo
Contributor
tubbo commented Jan 10, 2013

@jaywink that's what I meant the Loomio for, discussing how we're gonna do this. Just wanted to make sure we actually did want to do this before assuming you guys know what I'm thinking :)

@dskvr
dskvr commented Jan 10, 2013

@tubbo Thanks for your response, means a lot. @jaywink The truth, gotta love the truth. If I find myself in a position to dedicate time to this feature, I will. Unfortunately, hobby-projects have been on the backburner for years :-(

Note: I would put my two-cents on loomeo, but it's closed to registrations. Not that it matters too much since I have little to no say in the 'development priorities' of Diaspora considering I have made 0 commits to its' repo.

@jhass
Member
jhass commented Jan 10, 2013

Like over 90% of the people there... Just get your email address to anyone of us an you're in.

@ccverg
ccverg commented Mar 1, 2013

I just posted a $50 bounty that'll go to the person whose pull request gets accepted for this. I'd love to see this added, it's essential to the project...

Everyone that +1'd should consider adding to the bounty, we could make it worthwhile for a dev:

https://www.bountysource.com/#repos/diaspora/diaspora/issues/908

@Gimeit
Gimeit commented Mar 1, 2013

Yeah, what the hell. I added 10 bucks to the bounty - hopefully somebody will pick it up.

@tubbo
Contributor
tubbo commented Mar 1, 2013

Hey guys, were deliberating how to best go about this right now on the Diaspora Loomio! So you better have enough cash for the whole code team ;-)

-T

On Feb 28, 2013, at 8:13 PM, Gimeit notifications@github.com wrote:

Yeah, what the hell. I added 10 bucks to the bounty - hopefully somebody will pick it up.


Reply to this email directly or view it on GitHub.

@Flaburgan
Collaborator

Just put your emails here and we'll invite you to join the discussion

@creatinglake

Can I join the Loomio group?

I am seriously grateful that a group of developers are pushing Diaspora
forward. I really think its massively important for society in the
long-run.

Thanks!

Reply to this email directly or view it on GitHubhttps://github.com/diaspora/diaspora/issues/908#issuecomment-14277094
.

Adam Lake
540-585-4444

@DeadSuperHero
Member

Sure thing, Adam. Just send me an email address you'd like to use to
sean@joindiaspora.com, and I'll get you set up momentarily. :)

On Friday, March 1, 2013, Adam Lake wrote:

Can I join the Loomio group?

I am seriously grateful that a group of developers are pushing Diaspora
forward. I really think its massively important for society in the
long-run.

Thanks!

Reply to this email directly or view it on GitHub<
https://github.com/diaspora/diaspora/issues/908#issuecomment-14277094>
.

Adam Lake
540-585-4444


Reply to this email directly or view it on GitHubhttps://github.com/diaspora/diaspora/issues/908#issuecomment-14316977
.

@BitPuffin

So how is this coming along?

@Raven24
Member
Raven24 commented Jul 4, 2013
@gordon-morehouse

Dudes. Adding YET MORE barriers to entry (I have to join Loomio now? WTF is Loomio. I don't want to join more crap to talk about crap I'm already talking about here) is not helpful.

I'm stuck on a pod with an outdated version of Diaspora and no other-platform export plugins. I can't easily move. Given the entire point of Diaspora and the fact that it was a core feature, that's a problem.

"Incidentally, running your own pod is still the easiest way to be a part of DIASPORA."

That's also a problem. It "only" requires Rails and command-line experience! That is a 20-mile high barrier to entry to the average person. Hell, I've been a developer for 20 years and it's still something I'm unwilling to do because I don't have the time, money, or server space to do it (partly due to the choice of Rails, but that ship has LONG sailed) and I know precisely jack and squat about Ruby or Rails, having come from the Python and ::hrk:: PHP worlds.

Diaspora-type censorship resistant networks might be critical to humanity's future, but they will remain unused unless developers MAKE IT EASY.

No, I am not joining yet another goddamn platform to say that. If somebody wants it on Loomio, paste it please.

@gordon-morehouse

...and the friggin' Bountysource URL is dead so I can't donate to any bounty, which I am prepared to do despite having not enough money for cat litter, in case anybody's questioning deeds vs words. :p

@JosephBeck

Let's be honest, Diaspora* has dropped the ball and it isn't going to be going anywhere now. Any hype is dead and by the time they get their crap together, there will be a newer and better alternative. It's quite sad, but I've all but given up hope in this project. It has been years and it is STILL lacking this. Not having one of your main selling points / advertised features is a colossal failure. Maybe one day Diaspora* will make some worthy progress to make it a contender to things like Facebook and Google+. That day is not any time soon it seems.

Doesn't matter where the discussion is held, nothing is happening.

@Flaburgan
Collaborator

@gordon-morehouse here is the up-to-date bug-bounty ticket: https://www.bountysource.com/issues/59067-seed-migration-wizard which has already $60 on it.

We're all aware that diaspora* is not easy at the moment. That's why we're keeping 0. in front of the version number, to say it's stable, but not really ready to be deployed /maintained / updated easily. We just need more time, money and people working on it. But trust us (and join us ;)) we're going slowly but we're going the right way.

@JosephBeck a project is alive if people is believing in it. And there is still people believing in it ;)

@gordon-morehouse

Thanks for updating the bug bounty link. I will add a few bucks when I can, probably not much though. :/ I may have found another bug where I can contribute despite not knowing Ruby, at least.

On Sat, 01 Nov 2014 11:46:47 -0700, Fla notifications@github.com wrote:

@gordon-morehouse here is the up-to-date bug-bounty ticket: https://www.bountysource.com/issues/59067-seed-migration-wizard which has already $60 on it.

We're all aware that diaspora* is not easy at the moment. That's why we're keeping 0. in front of the version number, to say it's stable, but not really ready to be deployed /maintained / updated easily. We just need more time, money and people working on it. But trust us (and join us ;)) we're going slowly but we're going the right way.

@JosephBeck a project is alive if people is believing in it. And there is still people believing in it ;)


Reply to this email directly or view it on GitHub:
#908 (comment)

@Flaburgan
Collaborator

@gordon-morehouse we recently launched a "big" pod in France: https://framasphere.org (which already has 6000 users in only one month). We want to create a new community so we will probably be able to raise some money and improve diaspora*. The pod migration is one of the features we want to work on. Still many things to do, but hopefully we will success to do something ;)

@jhass jhass added the bounty label Nov 16, 2014
@sophiaphillyqueen

+1 ---- a very important feature, without which one can't take full advantage of Distributed Social Networks.

Oh --- and it needs to include (among other things) not just the ability to migrate directly --- but also the ability to backup your account yourself ---- just in case something goes a way you don't like in your pod, and you don't have enough foreknowledge to successfully abandon the ship.

@goobertron
Contributor

it needs to include (among other things) not just the ability to migrate directly --- but also the ability to backup your account yourself

Account data export and image export both work again with the release of 0.5.0.0.

@sophiaphillyqueen

Thank you for adding the feature. :-)

@unity100

+1 One of the most important features that must be implemented post haste

@jhass
Member
jhass commented Oct 11, 2015
  • What about private messages/conversations? So no full backup possible? What I liked about the old proposal is that almost all data can be retained and that I don't have to decide on my new home before the disaster happens. I could even setup my own pod after my old pod went down if I have downloaded my export/backup.
  • Why is being a backup location mandatory to being a move target? Is it really worth to tangle these two concepts so deeply?
  • Generating a new key doesn't have to be delayed, the moved message shall contain the new key and be the only message originating from the new pod that's signed with the old key. If a pod doesn't get the move message it won't accept content originating from the new pod anyhow since the updated diaspora* id will mismatch. I think that the old pod is revoked all authority over my profile asap is important too.
  • The mail address is insufficient as a network wide identifier. What happens if I store a "backup" from two pods to the same pods from accounts using the same mail address?
  • What about pictures? Download from the old pod if still up?
@jaywink
Contributor
jaywink commented Oct 12, 2015

What about private messages/conversations? So no full backup possible? What I liked about the old proposal is that almost all data can be retained

I think if we want to support a "full backup" then it has to be through some kind of import process, like we already have an issue for and what you mention and link the spec for. But IMHO the problem of dying pods is larger than the problem of moving pods. Users must be protected by automatic backup and fault tolerance. Not everybody is a techie that will remember to download their export every week. I think the back/restore process makes the network hugely more fault tolerant. It doesn't just allow moving pods, the point is to protect identities.

and that I don't have to decide on my new home before the disaster happens. I could even setup my own pod after my old pod went down if I have downloaded my export/backup.

Nowhere in the proposal it says users have to decide anything - the network takes care of that. But they have the option of choosing also, if they want to do simply a move without any disaster forcing them to. They can always move from their new location after a disaster.

Why is being a backup location mandatory to being a move target? Is it really worth to tangle these two concepts so deeply?

To reduce the amount of code and to keep it more simple, yes. Of course the migration process could be outside the restore process, but why? The end result is the same. If we want a separate "import and migrate" process as documented in your link, of course that could still be done, and they could share some of the same code. The the export/import solution does not make the network more fault tolerant.

Generating a new key doesn't have to be delayed, the moved message shall contain the new key and be the only message originating from the new pod that's signed with the old key. If a pod doesn't get the move message it won't accept content originating from the new pod anyhow since the updated diaspora* id will mismatch. I think that the old pod is revoked all authority over my profile asap is important too.

Sure, this makes things even more simpler. I'd still store the old key though, or what do you think about the "moved flag" logic regarding pods that missed the moved message?

The mail address is insufficient as a network wide identifier. What happens if I store a "backup" from two pods to the same pods from accounts using the same mail address?

Good point, so maybe the handle should be the unique key for the backups. I still think email validation on restore is super important.

What about pictures? Download from the old pod if still up?

They are not a part of the proposal due to the amount of space it would take to store the pictures on another pod. If the old pod is still alive the new pod could fetch all the pictures I guess? Or if we have an import process, the user could always import them to their new account. But I don't think the automatic backup/restore process has much room for pictures. The point in the proposal is to ensure identities survive, not all data.

@jhass
Member
jhass commented Oct 12, 2015

Good point, so maybe the handle should be the unique key for the backups. I still think email validation on restore is super important.

Just include the mail address inside the encrypted part of the backup and use the old diaspora* id as identifier, yes. Bonus point: the mail address isn't shouted out to other pods until the user gives consent to decrypt the backup.

Moved flag should be fine, but could be something for a later iteration and shouldn't be a requirement to a first implementation.

I don't see why this has to take two code paths even. We will need some "identity" tarball that has the necessary information to declare a new home towards the network and restore the base profile, most importantly the contacts. That encrypted base tarball could be included in the user export as well as automatically backed up. The import codepaths would only be different up to the point of having obtained that tarball and handing it to the user identity migrator. The importer of the user uploaded data could then simply proceed with importing other present data while the automatic backup importer would be done by then.

If you say the target location is chosen completely automatic, how is the user informed of the backup locations after disaster? It'll be displayed after choosing the backup passphrase and has to be remembered for eternity then?

@jaywink
Contributor
jaywink commented Dec 20, 2015

Just include the mail address inside the encrypted part of the backup and use the old diaspora* id as identifier, yes. Bonus point: the mail address isn't shouted out to other pods until the user gives consent to decrypt the backup.

Updated spec.

Moved flag should be fine, but could be something for a later iteration and shouldn't be a requirement to a first implementation.

Agree.

I don't see why this has to take two code paths even. We will need some "identity" tarball that has the necessary information to declare a new home towards the network and restore the base profile, most importantly the contacts. That encrypted base tarball could be included in the user export as well as automatically backed up. The import codepaths would only be different up to the point of having obtained that tarball and handing it to the user identity migrator. The importer of the user uploaded data could then simply proceed with importing other present data while the automatic backup importer would be done by then.

Added manual export / upload possibility. I don't however see why the backup export should be encrypted in this case. Having it unencrypted on the users hard drive of course would be bad but then, there is no point in encrypting it on the origin pod. The download process could inform user to handle backup storage carefully. However, of course, this means the users private key is in the archive, for the restore path. Do we want to take that route?

If you say the target location is chosen completely automatic, how is the user informed of the backup locations after disaster? It'll be displayed after choosing the backup passphrase and has to be remembered for eternity then?

There are two parts:

  • User is sent an email right after original backup is done by the pod being backed up from
  • User is sent an email right after receiving the first backup on the pod being backed up to

Yes, user would have to not only keep this information in store, but also remember the passphrase. We could do reminder emails every half a year or something. But the passphrase the user must store - that cannot be avoided?

Something that just came to mind, due to the manual procedure. We should probably require the user trying to restore is known to the pod. Otherwise anyone could upload any archive claiming to be someone from a dead pod or even never existed pod by just including a generated key pair. In the same way, we should verify the uploaded private key against the stored profile public key.

@jaywink
Contributor
jaywink commented Dec 20, 2015

Also, btw, of course, if we have a manual archive restore option, that would allow importing posts and conversations. I still wouldn't want this in the automated backups though.

Photos are also tricky, as mentioned in the linked spec. Maybe allow mass importing them from users separate downloaded archive or some kind of "pull" from still active pod as an option. But not in the regular archive imho.

@cmrd-senya
Contributor

The feature is being crowdfunded. Hope for your support :)

@cmrd-senya
Contributor

This message doesn't add anything new to what I have posted earlier on other channels (IRC, diaspora, wiki). However, since this is the home page for the issue and it's technical side, this must be referenced here.

Here is my spec for the issue solution. @jaywink's WIP spec was more focused on automatic backup/restore feature, while, I believe this is the last part of the problem. Before implementing automatic backup/restore, we must introduce protocol changes and the restore feature itself. Manual restore doesn't require any other changes, while automatic restore requires some additional features so it's natural to implement manual restore and only after that add the automatic backup/restore capability. Therefore, my spec doesn't have details on the automation, but on all the other things I described.

I have already received the review of the spec from @jhass. I'm waiting for a review by @denschub to move forward with the implementation.

Also, for the record, here are (currently WIP) PRs that are related to this issue resolution: #6726, #6750.

@dcapeletti

Someone knows what the state of development of this feature. 0%, 20%, 50%, 70% 100%

@SuperTux88
Member

see last comment from senya, he is working on it, but there is no exact %-measurement for it.

@Bugsbane
Bugsbane commented Aug 8, 2016

Last I heard, I think he was waiting for a couple of issues to be fixed elsewhere that his work depended on, and he was trying to work on some of his secondary improvements while he waited.

@Timoses
Timoses commented Aug 26, 2016

To me it seems that selecting only one pod could be problematic. What if a businessman wants to use diaspora* and often travels between Europe and US. So when he selects a US pod he'll have lag from Europe, and vice versa.

Considering migration stuff etc. wouldn't it make sense that a person can manage the pods that she/he is using? For example: I know I'm often traveling between Japan, Europe and US. I'd like my data to be on pods in each of those areas so I won't suffer lag. Subsequently, data would be synced between these selected pods. This also guarantees some resilience in case one server dies.

There comes into play another problem: Logging into your account. A great feature would be to be able to login to your pod from one website (which knows which email belongs to what pod(s) somehow) and automatically routes you to the correct pod. Otherwise you'd need to switch between different websites all the time -> annoying. And anything that is annoying will make diaspora less attractive to use.

A further connected issue: I just signed in to diaspora and saw that there's an e-mail with ones name attached to the pod. Not sure what it is used for actually, but what is that e-mail worth if it's going to change when migrating to another pod (or using multiple pods)?

So what I'm suggesting is a framework to allow multiple pods to be selected by users and synced between each other. A one-point login to your correct pod (depending on your location?) would help to mitigate annoying domain-switching.

@cmrd-senya
Contributor

@Timoses, you enumerated a number of issues, and I feel that they are loosely related to the migration feature. By design diaspora currently requires a user to have an account on the home pod which is an entry point to the diaspora network. Changing this assumption would require reconsideration of many design points and it is non-obvious. Introduction of serious design changes requires a strong reason. Pod access time of some frequently travelling person doesn't seem to me serious enough to justify the network design shift. If this is an real issue for someone, a more appropriate and less radical solution may be worked out.

Also, please note, that conceptual discussions and feature proposals are preffered to be discussed on Loomio. If you come up with a proposal on how to improve design of the diaspora federation, feel free to share it for the discussion on Loomio first. GitHub is a place for technical discussions (e.g. how do we implement feature X that was approved on Loomio before).

Thank you for your participation!

@unity100

biggest issue with there being no pod migration is that if the pod i
am in goes bust, my account, connections, everything go bust.

that practically kills all the purpose of decentralization.

as long as i am locked into one pod, i am on a centralized network.
content spreading in a decentralized fashion does not make up for it.

as appalling as it is, i can trust that my account on facebook will be
there for a loooong time. even if they use my data as a means to
profit off of me by selling it to government and advertisers, that
also ensures continuity of my account.

but with diaspora, i am obliged to trust the effort of a small group
of people or even one person setting up a server somewhere and then
being able to maintain it through the years. which can go bust at any
time due to a million reasons.

if, somehow, an established organization with endurance sets the
server up (ie a corporation, a ngo etc) i have to be wary of their
motives. yeah that's a great diaspora server with tens of thousands of
users, but it can even be a front by an intelligence organization to
spy on their users - they do it with tor.

or i can set up my own server, and have to deal with everything that
comes with setting up and maintaining a server. which, many of
diaspora users cannot do.

or i can use the apparently not so mature p2p client and battle with
many issues.

............

OR, we can have pod migration and forget all of these important issues.

On 8/26/16, Senya notifications@github.com wrote:

@Timoses, you enumerated a number of issues, and I feel that they are
loosely related to the migration feature. By design diaspora currently
requires a user to have an account on the home pod which is an entry point
to the diaspora network. Changing this assumption would require
reconsideration of many design points and it is non-obvious. Introduction of
serious design changes requires a strong reason. Pod access time of some
frequently travelling person doesn't seem to me serious enough to justify
the network design shift. If this is an real issue for someone, a more
appropriate and less radical solution may be worked out.

Also, please note, that conceptual discussions and feature proposals are
preffered to be discussed on
Loomio. If you come up
with a proposal on how to improve design of the diaspora federation, feel
free to share it for the discussion on Loomio first. GitHub is a place for
technical discussions (e.g. how do we implement feature X that was approved
on Loomio before).

Thank you for your participation!

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#908 (comment)

@creatinglake

No seed migration defeats the point of a decentralized social network. If
you can't migrate your data without loosing your social connections it's
not really decentralized. I an keeping my eyes on this, solidplatform.org

On Fri, Aug 26, 2016 at 10:17 AM, unity100 notifications@github.com wrote:

biggest issue with there being no pod migration is that if the pod i
am in goes bust, my account, connections, everything go bust.

that practically kills all the purpose of decentralization.

as long as i am locked into one pod, i am on a centralized network.
content spreading in a decentralized fashion does not make up for it.

as appalling as it is, i can trust that my account on facebook will be
there for a loooong time. even if they use my data as a means to
profit off of me by selling it to government and advertisers, that
also ensures continuity of my account.

but with diaspora, i am obliged to trust the effort of a small group
of people or even one person setting up a server somewhere and then
being able to maintain it through the years. which can go bust at any
time due to a million reasons.

if, somehow, an established organization with endurance sets the
server up (ie a corporation, a ngo etc) i have to be wary of their
motives. yeah that's a great diaspora server with tens of thousands of
users, but it can even be a front by an intelligence organization to
spy on their users - they do it with tor.

or i can set up my own server, and have to deal with everything that
comes with setting up and maintaining a server. which, many of
diaspora users cannot do.

or i can use the apparently not so mature p2p client and battle with
many issues.

............

OR, we can have pod migration and forget all of these important issues.

On 8/26/16, Senya notifications@github.com wrote:

@Timoses, you enumerated a number of issues, and I feel that they are
loosely related to the migration feature. By design diaspora currently
requires a user to have an account on the home pod which is an entry
point
to the diaspora network. Changing this assumption would require
reconsideration of many design points and it is non-obvious.
Introduction of
serious design changes requires a strong reason. Pod access time of some
frequently travelling person doesn't seem to me serious enough to justify
the network design shift. If this is an real issue for someone, a more
appropriate and less radical solution may be worked out.

Also, please note, that conceptual discussions and feature proposals are
preffered to be discussed on
Loomio. If you come
up
with a proposal on how to improve design of the diaspora federation, feel
free to share it for the discussion on Loomio first. GitHub is a place
for
technical discussions (e.g. how do we implement feature X that was
approved
on Loomio before).

Thank you for your participation!

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#908 (comment)


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#908 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA70qF-xROxxF39JR9XJyEaCs0BOI53kks5qjvVvgaJpZM4ABDuW
.

Adam Lake
540-285-0083

@goobertron
Contributor

As cmrd-senya said, please reserve Github for discussion of specific technical issues and how to solve them. Please take general discussion about the software and/or project on to Diaspora itself, or contribute to discussions on the project's decision-making platform, Loomio. Here are links to two existing discussions on this issue: Backup and Restore / Personal Data Import/Export.

@creatinglake

sounds good goob. Sorry.

On Fri, Aug 26, 2016 at 2:19 PM, goob notifications@github.com wrote:

As cmrd-senya said, please reserve Github for discussion of specific
technical issues and how to solve them. Please take general discussion
about the software and/or project on to Diaspora itself, or contribute to
discussions on the project's decision-making platform, Loomio. Here are
links to two existing discussions on this issue: Backup and Restore
https://www.loomio.org/d/qsVJ2K1t/backup-and-restore / Personal Data
Import/Export
https://www.loomio.org/d/w6nfQPFc/personal-data-import-export.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#908 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA70qJoYlCE3Zv8QqQBoAc6JBr6EEP8bks5qjy46gaJpZM4ABDuW
.

Adam Lake
540-285-0083

@unity100

im also copying my post to loomio then.

On 8/26/16, Adam Lake notifications@github.com wrote:

sounds good goob. Sorry.

On Fri, Aug 26, 2016 at 2:19 PM, goob notifications@github.com wrote:

As cmrd-senya said, please reserve Github for discussion of specific
technical issues and how to solve them. Please take general discussion
about the software and/or project on to Diaspora itself, or contribute to
discussions on the project's decision-making platform, Loomio. Here are
links to two existing discussions on this issue: Backup and Restore
https://www.loomio.org/d/qsVJ2K1t/backup-and-restore / Personal Data
Import/Export
https://www.loomio.org/d/w6nfQPFc/personal-data-import-export.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#908 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA70qJoYlCE3Zv8QqQBoAc6JBr6EEP8bks5qjy46gaJpZM4ABDuW
.

Adam Lake
540-285-0083

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#908 (comment)

@denschub
Member

(this is c&p'ed from an email I just sent, but it's smarter to have that discussion here on GitHub. Unfortunately, Gist does not send notification for new comments, so that's meh.)

Hi Senya,

please find my feedback to your specification below. Thank you very much for your work. It is unclear to me if you intend to use this document for diaspora only or if you plan on having a general migration definition for other social networks as well. If it's the latter, this spec misses a lot of details.

(I added this paragraph after writing everything else). We probably should clear that first. What do you wanna do here? Initially, we talked about a migration process that would work for other networks as well. This document looks like a plan of what diaspora needs to do, which is not the same. I just read the entire thing and I still don't have an idea on how this is supposed to work in detail.

Please put the "process description", that is, the needed steps first, followed by the details. Take a look at the OpenID flow examples for example.

There is no need to differentiate between "frontend" and "backend" steps here. Write the directions as unbiased, simple, and neutral as possible. For example (just wrote that, this is not the actual flow.)

  1. User initiates the account migration process by entering the new servers URL.
  2. The old server sends a AccountMoveRequested to the new server.
  3. The new server sends a Profile to the old server.
  4. After the old server imported the profile, an user archive is created and a link to the download is provided to the user.
  5. The user uploads the archive to the new server.
  6. The new server sends a AccountImportCompleted message to the old server.
  7. The old server deletes the user account and marks the profile as moved.

And so on.

Here is the original feedback, but I guess we should define our goals first...

This document is intended to make Dennis happy

I hope that this was meant to be as a joke. If not, I sincerely am disappointed that I am the only one seeing the reason for having a very complex and very risky featured and its underlying principles spec'ed and discussed.

and which can talk to each other

Nit: "communicate with each other"

TODO: Do we want to support "account merging"? [...] Technically it doesn't differ from the migration to a new account and can be useful.

Are you sure that there is no difference? I imagine merging data created by two different keys with different signatures being quite edgy.

Federation entity

Other implementations will not read our source to understand the resulting XML. Please add the actual XML that has to be send on user moves or we'll never see anyone else being compatible with our stuff.

# the old account id
property :author

I assume this is talking about the "account ID" (or, as we know it, the diaspora handle)?

Someone tries to discover a user by the old ID

Please specify that a permanent redirect should be used, just to be sure.

If that happens, the old home pod must send the AccountRename entity to the pod from where the message has arrived.

I would also suggest relaying of the message, received by the old home pod to the new home pod, but I'm not sure that the additional stability would worth the possible complexity of the implementation.

If we're able to determinate that the user moved right in the receive step, we could return an error code and the pod would automatically retry. But I doubt we'd be able to do this. :/

New home pod only: create a new user and person in the pod's DB

How does the new pod get the users details like profile information, email addresses? What about the users password? Should we send a password recovery email immediately after the account creation?

Tombstone old person and profile

"Tombstone" is not a verb and even if it would be, it's ambiguous. Please specify that the origin pod has to remove all user contents and the profile. Step seems duplicated with step 3. Also, link to GitHub is not permanent, press the y key on GitHub to make it reload with an URL that contains the latest revision.

Update all the DB references

What are "all" references? Where is the person used? How does the old pod get the new profile after it's created (do we fetch?). After all, most entities are related to a person which we don't know about after some federation happened.

The implementation of this method is postponed to the point after the manual export is implemented.

Wondering about the timeline. Alice starts the migration from Source to Target. When does the import happen? When does the migration happen? Is the archive import a step that can be done after the account is moved? If so (as specified later in the document), how can "The archive owner's account is not closed" be a validation argument?

I am totally lost about the actual import stap. In "Backend migration process" you write that the ID change procedure needs to be done before importing the archive. However, the archive includes the users details, which we need to create the new account. I fear we deadlock'ed here.

@cmrd-senya
Contributor
cmrd-senya commented Sep 26, 2016 edited

Hi! Thank you for your review. I'm currently restructuring my document in order to correct the issues you described. I'll update my gist ASAP.

@cmrd-senya
Contributor
cmrd-senya commented Sep 28, 2016 edited

I've updated the gist. I've changed the document structure and I hope it is more clear now and changes consider your hints.

We probably should clear that first. What do you wanna do here? Initially, we talked about a migration process that would work for other networks as well. This document looks like a plan of what diaspora needs to do, which is not the same.

I updated the "scope" section to clarify it. The main goal is to describe account migration for diaspora*. Less important goal, but still a considered one is support of the feature by other compatible implementations. So sections 2-5 are mostly implementation invariant, though still not detailed enough, but nevertheless useful for those who want implement a compatible implementation. Sections 6-8 are purely diaspora* specific.

Please put the "process description", that is, the needed steps first, followed by the details. Take a look at the OpenID flow examples for example.

There is no need to differentiate between "frontend" and "backend" steps here. Write the directions as unbiased, simple, and neutral as possible.

I've introduced "Seed migration flow" section.

This document is intended to make Dennis happy

I hope that this was meant to be as a joke. If not, I sincerely am disappointed that I am the only one seeing the reason for having a very complex and very risky featured and its underlying principles spec'ed and discussed.

You don't like my jokes too much, do you? :) Sure, don't take that seriously, I realize the importance of spec&discussion.

TODO: Do we want to support "account merging"? [...] Technically it doesn't differ from the migration to a new account and can be useful.

Are you sure that there is no difference? I imagine merging data created by two different keys with different signatures being quite edgy.

Ok, if it doesn't seem trivial, then this idea discussion must be put aside at least before we have more urgent things done.

Federation entity
Other implementations will not read our source to understand the resulting XML. Please add the actual XML that has to be send on user moves or we'll never see anyone else being compatible with our stuff.

I only posted XML message example, hope this is enough for a while. I can add some description on XML message structure or add an XML schema, however it looks pretty self-descriptive.

# the old account id
property :author

I assume this is talking about the "account ID" (or, as we know it, the diaspora handle)?

Right, diaspora_federation is moving towards using author as a field name for this matter.

New home pod only: create a new user and person in the pod's DB

How does the new pod get the users details like profile information, email addresses? What about the users password? Should we send a password recovery email immediately after the account creation?

It was described in the "UI for the account migration" section. After a user have provided an archive to the new pod he is forwarded to a page with a form similar to our present registration form, with the difference that email address is prefilled with the value from the archive. Also, profile information is being imported from the archive as was described in the "Archive import sequence" section.

"Tombstone" is not a verb and even if it would be, it's ambiguous. Please specify that the origin pod has to remove all user contents and the profile. Step seems duplicated with step 3.

I drew a block diagram to make the process more detailed, it is in the "Updating DB state of a diaspora* pod when performing ID change" section now. There is no point in removing "all user contents", because some of the content have relation to other people, and will be rereferenced to the new identity anyway. Only some of the user content may be removed from the old home pod, e.g. contacts and profile data.

What are "all" references? Where is the person used? How does the old pod get the new profile after it's created (do we fetch?). After all, most entities are related to a person which we don't know about after some federation happened.

Ditto, see "Updating DB state of a diaspora* pod when performing ID change" section. The old pod get the new profile just like any other pod, by receiving the AccountRename message.

Wondering about the timeline. Alice starts the migration from Source to Target. When does the import happen? When does the migration happen? Is the archive import a step that can be done after the account is moved? If so as specified later in the document

It's unclear why you say "archive import a step that can be done after the account is moved ... as specified later in the document". There is no such claims neither in the previous version of the spec, nor in the new. Archive import is performed along with the moving process, it may begin just after the new user record has been created, and the AccountRename federation message is only sent when the archive is already imported.

how can "The archive owner's account is not closed" be a validation argument?

Archive is validated before the migration process ever started.

I am totally lost about the actual import stap. In "Backend migration process" you write that the ID change procedure needs to be done before importing the archive. However, the archive includes the users details, which we need to create the new account. I fear we deadlock'ed here.

I see no deadlocks here. We read the archive when the user provides it at the very beginning and use data from it to create a new account. The ID change doesn't depend on the archive data, except of the step where the federation message is sent to the contacts, which can easily be postponed until the import is finished. This very sequence was described in the previous version as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment