Seed Migration Wizard #908

Open
diaspora-redmine-github-migration opened this Issue Jul 8, 2011 · 75 comments
@diaspora-redmine-github-migration

Issue 505 from bugs.joindiaspora.com
Created by: Wavesonics
On Mon Nov 1 20:03:57 2010

Priority: Normal
Status: New

There should be a wizard in the Account UI for migrating your seed from one pod to the another, all server to server.

This way contacts could be notified of the move in a secure way, and the old account could be deleted when the move was complete.

There is a $200 open bounty on this issue. Add to the bounty at Bountysource.

@diaspora-redmine-github-migration

Comment by: Hexagon
On Mon Nov 1 20:45:46 2010

+1

@diaspora-redmine-github-migration

Comment by: Cathryne
On Fri Nov 26 18:43:11 2010

+1 with a user-defined overlap time until the old account automatically closes.

@luzvioleta

This is sincerely critical .... any idea on the state of this ? (or the partial photo & xml export ? )

@Nygu

It's not being worked on yet. I guess it'll come somewhere during beta. On the one hand I think it's a shame, on the other hand I'm glad they are working on more usable things first.

@ghost

+1

@pniedzielski

I'm willing to work on this with someone else. Have a good amount of programming experience, but just learned Ruby...also, due to classes, unable to work on it every day.

Anyone else willing to help with this?

@sebastianbuettrich

If there s need for it, i d be willing to help with this:

relevant experience:
databases, replication, migration (12 yrs mysql, 15 yrs general sql), db architecture,
coding: C, perl, php

no skills in ruby at all :(

@pravi
diaspora* social network member
@pniedzielski

I've sent ideas to diaspora-dev in this mail thread, thanks for the discussion, guys!

@sebastianbuettrich

great. i ll gladly contribute where it s needed -
i m blank on ruby though.

my feeling is that we ll have to look a little bit at database structure, redundancies, in the process -
though you rightfully say it s out of scope here -

but it s something i ll gladly comment on / help with.

@Wavesonics

Any progress on this? Seems like an important feature before going to beta.

@pniedzielski

It's not on the beta checklist, and though it is an important feature, I don't think it need be -- Diaspora's built on an iterative development process, so "Beta" is much more fluid (and possibly having less substantial meaning) here than it is in other sorts of Free Software projects. That last sentence is my opinion only, so it's fine if you feel differently.

More directly, I'm trying to learn the internals of Diaspora (strong documentation is...lacking), before I attempt this, as well as coordinating a release for my own Free Software Project in the next two weeks. I plan to attack it around Thanksgiving (US one, three weeks away). That's my approximate timeframe.

@stephens2424

I'm not sure if or when I'd be able to help out on this, but whoever works on this, it would be ideal if the system could take care of updating all your followers/friends of the move for you. Transparently, too, so the remote pods will just receive an update when they next check for information. It just seems important that this doesn't recreate the hassle of changing email addresses or phone numbers.

@zeorin

Perhaps it would be a good idea if there is functionality to not only move accounts, but in the same process to merge accounts...
And perhaps to sync more than one instance of an account between two or more servers: for redundancy purposes.

@twoolie

Would the move generate a HTTP 301 Permanent Redirect to the new page location, or would the old pod mirror the updates from the new pod?

@zeorin

That would depend on whether one moved or merged the account (301) or synced them (mirror). Although you should be able to log into either pod if you're syncing.

@jhass
diaspora* social network member

Follow up #2616

@rriemann

Hey, is this really abandoned? You advertised this feature on your very fist landing page:
http://diasporaproject.org/

@sysfu

I would like to see this feature implemented soon

@stragu

Yes, this would be great. What would happen if a pod closes down or doesn't update the version of Diaspora??

@darklajid

This issue is closed but linked to from

http://diasporaproject.org/#benefits

#2616 (mentioned as 'follow-up') is closed as well, so the only 'status' might potentially exist in a mailing list archive.

Wouldn't it make sense to update the home page to stop sending people here? I'd vote to remove that 'coming soon' feature, because it seems it's not. At the very least the link to this specific issue should be removed.

@creatinglake

This is essential for the long-term viability of the Diaspora mission; the whole point ti to have freedom of movement. Still, great job Diaspora HQ.

@rrohrer

This feature would be handy. Also since this is clearly dead, the "coming soon" should be removed from diasporaprojects home page.

@Raven24
diaspora* social network member

https://groups.google.com/d/topic/diaspora-discuss/zOpARcOPfoM/discussion is the accumulated feature discussion for this issue.
This is still high priority, but apart from many ideas we never really got any code for this...

@JosephBeck

So we can't move pods easily yet? We just have to delete our old account and make a new one? I thought this was going to be a main feature but seems its not completed. I see that I can export data but not import. Least on Diasp.org which I plan to move from soon.

@kqr

I've been waiting quite a while (years, by now) to sign up on a pod since it's unsure if I'll be able to move to a self-hosted pod later on. I was happy when I saw the "Coming soon" on the main page but apparently this is not something that's in the works. Is there any word on what the problem is? The UI? The code? The security?

@jaywink

Even a stupid migrate process would be cool to start off with - just to be able to import old posts (not trigger any federation for them) and do an automatic aspects creation and contact adding.

@ghost

It would be great to have this function! I myself for example would be very interested in migrating from the joindiaspora-pod to a pod here in Europe. I feel that a function like this would make it more convenient for a user to continue using diaspora when their pod may be shutting down. It would be simple and great just to migrate to another pod and keep on interacting like nothing happened. It is a decentralized idea going well together with the idea of the concept of Diaspora*.

@Gimeit

The lack of this feature is the only thing keeping me from signing up for and donating to Diaspora. I'm sad to see that its development has apparently been abandoned - seems like a pretty core concept.

@jaywink

I wouldn't say it has been abandoned - it is a constant discussion topic among not only users talking about features but also many devs that write code for Diaspora. You have to remember that no one writes code for Diaspora for a living any more - it's all volunteers. So this kind of big features will take time. It is a must feature that has to be done at some point though, sooner the better.

@rivendale2010

some facility for removal of dead handles would be good to include, maybe generate a webfinger on 404 to a guid or whatever :)

@JosephBeck

It is something as simple as an import feature for the data it lets us export, it isn't rocket science. Seed migration was one of their MAIN advertising points and it has yet to be implemented or even trial implemented. Diaspora* just isn't functional enough and uses very unpopular software among developers. It isn't worth many people's hassle. It did what it set out to do, proved you can have decentralized social networks. Problem is, it just doesn't have really any use. Even if they finished it not enough people even know about Diaspora* to matter. Tech. is an industry where you have to stay focused and working, if you fall too far behind you'll be left there and that is what has happened to Diaspora*. For me personally, it just isn't worth the time to set up and work on. Social Networks of all kinds are declining anyways as people lose faith in Facebook, have no use for Twitter, Google+ is only used by google fanboys and their friends, and things like this were never finished with the basic essentials.

@jaywink

Please keep the discussing issues on Github - this is not a discussion forum but a place to follow up on issues relating to Diaspora*. There is no point fretting about the missing feature - it will be implemented when some developer writes the code. All the devs do this on their spare time and without any compensation. If you really demand something to be implement it - do it or find someone to do it! That is the true power of open source :)

@JosephBeck

You seemed to have missed the point, it isn't just a missing feature. It is a flagship feature that has been used in advertising Diaspora* since the start. That is a major issue and should be given utmost attention and it isn't. YOU were discussing what someone else had said and making excuses, and I was doing the same by commenting on what you said and removing the excuses... You can't blame the fact it is volunteer, several volunteer software developers and even OS developers have brand new features working and testing on a daily basis. You can't state something as a flagship feature then ignore it.

@Gimeit

JosephBeck hit the nail on the head - from an end-user perspective, the ability to control your own data is THE distinguishing feature of Diaspora over other social networks. Without the option to migrate pods this feature doesn't exist. There is obviously a lot of other work being done on the project, but it's doomed to obscurity as long as the core functionality is crippled.

@jaywink

@JosephBeck @Gimeit no one is ignoring this feature, I don't know a single person who is somehow participating who is ignoring this or doesn't want it done :)

The reason this isn't implemented is not because it is not wanted but because there are a limited amount of devs working on the code and to be frank there are a lot of things to do and fix - this is only one of them. Recently the most active contributors have been concentrating on code cleanup since the whole community governance thing. But the truth is that more devs are needed to keep things rolling - so if you know some Ruby just jump in and help create an awesome open source social network :)

@jaywink

@DeadSuperHero just noticed this issue is closed for some reason - can you open it? :D

@Raven24 Raven24 reopened this Nov 16, 2012
@dskvr

Without this feature, this project is nearly worthless. I join Pod A, Pod A starts advertising, and changes their privacy policy. I no longer own my data. I delete Pod A and start with Pod B .... Seems pointless. No different than moving from Instagram to something else. Without this feature, Diaspora isn't Diaspora. This topic is 2 years old. Maybe another Kickstarter to get this feature done?

@tubbo

lol I love how every feature on this issue tracker is "urgent" and the project is totally "worthless" without it.

edit: Incidentally, running your own pod is still the easiest way to be a part of DIASPORA. You can own your data, but at this time you must be self-reliant to do so. Soon, we hope to incorporate this feature, but we want to make sure it's done in a safe and robust way.

@dskvr

I understand that, but it seems that this feature in particular negates the very core concept of Diaspora ... not everybody can run a pod, to assume so would be a serious flaw, and is nearly an admission of defeat. I am stuck on a pod with advertising, a shifting privacy policy and an old version of Diaspora. I cannot switch. If I was one of the 98% of the people in the world who don't do web development and have no empathy or understanding for what it takes, I would simply leave, and NEVER come back. I respect and admire everybody who contributes to this, and am grateful for any and all progress, but there has apparently been minimal progress on this incredibly important front.

@tubbo

@dskvr The only bottleneck in getting a pod up and running at this time is command-line and Rails experience. We're working on an app right now (might even be out there in beta) that will provision new pods on Heroku, so you basically click a button, enter in your Heroku account and API key, and the app does the rest of the work. The other leg of that is what I'm working on, an app that will tag the latest releases on Github and send an API call that redeploys your app. This way, we can actually institute a kind-of continuous integration system, making sure each of our pod servers are up to date all the time. In situations like when the Rails XML/mass-assignment vulnerabilities were exposed this week, it would be super cool if we could ensure security across our ENTIRE network, without needing root access to each box.

That being said, this is still an issue that needs dealing with. I don't think a Github Issue is the place to discuss how to best move forward on this, so I made a Loomio discussion: https://www.loomio.org/discussions/1653

If you're on our Loomio, please vote on the current proposal and if we get a majority amount of "No's", we'll discuss how to best implement this feature.

@jaywink

I really don't see any point in voting about this feature :) Who wouldn't want pod migration to happen?

The problem is no one has done it yet and all the developers are now doing this as a hobby. So until someone picks this up nothing much to debate (except technical ideas of course) :P

@tubbo

@jaywink that's what I meant the Loomio for, discussing how we're gonna do this. Just wanted to make sure we actually did want to do this before assuming you guys know what I'm thinking :)

@dskvr

@tubbo Thanks for your response, means a lot. @jaywink The truth, gotta love the truth. If I find myself in a position to dedicate time to this feature, I will. Unfortunately, hobby-projects have been on the backburner for years :-(

Note: I would put my two-cents on loomeo, but it's closed to registrations. Not that it matters too much since I have little to no say in the 'development priorities' of Diaspora considering I have made 0 commits to its' repo.

@jhass
diaspora* social network member

Like over 90% of the people there... Just get your email address to anyone of us an you're in.

@ccverg

I just posted a $50 bounty that'll go to the person whose pull request gets accepted for this. I'd love to see this added, it's essential to the project...

Everyone that +1'd should consider adding to the bounty, we could make it worthwhile for a dev:

https://www.bountysource.com/#repos/diaspora/diaspora/issues/908

@Gimeit

Yeah, what the hell. I added 10 bucks to the bounty - hopefully somebody will pick it up.

@tubbo
@Flaburgan

Just put your emails here and we'll invite you to join the discussion

@creatinglake
@DeadSuperHero
diaspora* social network member
@BitPuffin

So how is this coming along?

@Raven24
diaspora* social network member
@gordon-morehouse

Dudes. Adding YET MORE barriers to entry (I have to join Loomio now? WTF is Loomio. I don't want to join more crap to talk about crap I'm already talking about here) is not helpful.

I'm stuck on a pod with an outdated version of Diaspora and no other-platform export plugins. I can't easily move. Given the entire point of Diaspora and the fact that it was a core feature, that's a problem.

"Incidentally, running your own pod is still the easiest way to be a part of DIASPORA."

That's also a problem. It "only" requires Rails and command-line experience! That is a 20-mile high barrier to entry to the average person. Hell, I've been a developer for 20 years and it's still something I'm unwilling to do because I don't have the time, money, or server space to do it (partly due to the choice of Rails, but that ship has LONG sailed) and I know precisely jack and squat about Ruby or Rails, having come from the Python and ::hrk:: PHP worlds.

Diaspora-type censorship resistant networks might be critical to humanity's future, but they will remain unused unless developers MAKE IT EASY.

No, I am not joining yet another goddamn platform to say that. If somebody wants it on Loomio, paste it please.

@gordon-morehouse

...and the friggin' Bountysource URL is dead so I can't donate to any bounty, which I am prepared to do despite having not enough money for cat litter, in case anybody's questioning deeds vs words. :p

@JosephBeck

Let's be honest, Diaspora* has dropped the ball and it isn't going to be going anywhere now. Any hype is dead and by the time they get their crap together, there will be a newer and better alternative. It's quite sad, but I've all but given up hope in this project. It has been years and it is STILL lacking this. Not having one of your main selling points / advertised features is a colossal failure. Maybe one day Diaspora* will make some worthy progress to make it a contender to things like Facebook and Google+. That day is not any time soon it seems.

Doesn't matter where the discussion is held, nothing is happening.

@Flaburgan

@gordon-morehouse here is the up-to-date bug-bounty ticket: https://www.bountysource.com/issues/59067-seed-migration-wizard which has already $60 on it.

We're all aware that diaspora* is not easy at the moment. That's why we're keeping 0. in front of the version number, to say it's stable, but not really ready to be deployed /maintained / updated easily. We just need more time, money and people working on it. But trust us (and join us ;)) we're going slowly but we're going the right way.

@JosephBeck a project is alive if people is believing in it. And there is still people believing in it ;)

@gordon-morehouse
@Flaburgan

@gordon-morehouse we recently launched a "big" pod in France: https://framasphere.org (which already has 6000 users in only one month). We want to create a new community so we will probably be able to raise some money and improve diaspora*. The pod migration is one of the features we want to work on. Still many things to do, but hopefully we will success to do something ;)

@jhass jhass added the bounty label Nov 16, 2014
@sophiaphillyqueen

+1 ---- a very important feature, without which one can't take full advantage of Distributed Social Networks.

Oh --- and it needs to include (among other things) not just the ability to migrate directly --- but also the ability to backup your account yourself ---- just in case something goes a way you don't like in your pod, and you don't have enough foreknowledge to successfully abandon the ship.

@goobertron

it needs to include (among other things) not just the ability to migrate directly --- but also the ability to backup your account yourself

Account data export and image export both work again with the release of 0.5.0.0.

@sophiaphillyqueen

Thank you for adding the feature. :-)

@unity100

+1 One of the most important features that must be implemented post haste

@jhass
diaspora* social network member
  • What about private messages/conversations? So no full backup possible? What I liked about the old proposal is that almost all data can be retained and that I don't have to decide on my new home before the disaster happens. I could even setup my own pod after my old pod went down if I have downloaded my export/backup.
  • Why is being a backup location mandatory to being a move target? Is it really worth to tangle these two concepts so deeply?
  • Generating a new key doesn't have to be delayed, the moved message shall contain the new key and be the only message originating from the new pod that's signed with the old key. If a pod doesn't get the move message it won't accept content originating from the new pod anyhow since the updated diaspora* id will mismatch. I think that the old pod is revoked all authority over my profile asap is important too.
  • The mail address is insufficient as a network wide identifier. What happens if I store a "backup" from two pods to the same pods from accounts using the same mail address?
  • What about pictures? Download from the old pod if still up?
@jaywink

What about private messages/conversations? So no full backup possible? What I liked about the old proposal is that almost all data can be retained

I think if we want to support a "full backup" then it has to be through some kind of import process, like we already have an issue for and what you mention and link the spec for. But IMHO the problem of dying pods is larger than the problem of moving pods. Users must be protected by automatic backup and fault tolerance. Not everybody is a techie that will remember to download their export every week. I think the back/restore process makes the network hugely more fault tolerant. It doesn't just allow moving pods, the point is to protect identities.

and that I don't have to decide on my new home before the disaster happens. I could even setup my own pod after my old pod went down if I have downloaded my export/backup.

Nowhere in the proposal it says users have to decide anything - the network takes care of that. But they have the option of choosing also, if they want to do simply a move without any disaster forcing them to. They can always move from their new location after a disaster.

Why is being a backup location mandatory to being a move target? Is it really worth to tangle these two concepts so deeply?

To reduce the amount of code and to keep it more simple, yes. Of course the migration process could be outside the restore process, but why? The end result is the same. If we want a separate "import and migrate" process as documented in your link, of course that could still be done, and they could share some of the same code. The the export/import solution does not make the network more fault tolerant.

Generating a new key doesn't have to be delayed, the moved message shall contain the new key and be the only message originating from the new pod that's signed with the old key. If a pod doesn't get the move message it won't accept content originating from the new pod anyhow since the updated diaspora* id will mismatch. I think that the old pod is revoked all authority over my profile asap is important too.

Sure, this makes things even more simpler. I'd still store the old key though, or what do you think about the "moved flag" logic regarding pods that missed the moved message?

The mail address is insufficient as a network wide identifier. What happens if I store a "backup" from two pods to the same pods from accounts using the same mail address?

Good point, so maybe the handle should be the unique key for the backups. I still think email validation on restore is super important.

What about pictures? Download from the old pod if still up?

They are not a part of the proposal due to the amount of space it would take to store the pictures on another pod. If the old pod is still alive the new pod could fetch all the pictures I guess? Or if we have an import process, the user could always import them to their new account. But I don't think the automatic backup/restore process has much room for pictures. The point in the proposal is to ensure identities survive, not all data.

@jhass
diaspora* social network member

Good point, so maybe the handle should be the unique key for the backups. I still think email validation on restore is super important.

Just include the mail address inside the encrypted part of the backup and use the old diaspora* id as identifier, yes. Bonus point: the mail address isn't shouted out to other pods until the user gives consent to decrypt the backup.

Moved flag should be fine, but could be something for a later iteration and shouldn't be a requirement to a first implementation.

I don't see why this has to take two code paths even. We will need some "identity" tarball that has the necessary information to declare a new home towards the network and restore the base profile, most importantly the contacts. That encrypted base tarball could be included in the user export as well as automatically backed up. The import codepaths would only be different up to the point of having obtained that tarball and handing it to the user identity migrator. The importer of the user uploaded data could then simply proceed with importing other present data while the automatic backup importer would be done by then.

If you say the target location is chosen completely automatic, how is the user informed of the backup locations after disaster? It'll be displayed after choosing the backup passphrase and has to be remembered for eternity then?

@jaywink

Just include the mail address inside the encrypted part of the backup and use the old diaspora* id as identifier, yes. Bonus point: the mail address isn't shouted out to other pods until the user gives consent to decrypt the backup.

Updated spec.

Moved flag should be fine, but could be something for a later iteration and shouldn't be a requirement to a first implementation.

Agree.

I don't see why this has to take two code paths even. We will need some "identity" tarball that has the necessary information to declare a new home towards the network and restore the base profile, most importantly the contacts. That encrypted base tarball could be included in the user export as well as automatically backed up. The import codepaths would only be different up to the point of having obtained that tarball and handing it to the user identity migrator. The importer of the user uploaded data could then simply proceed with importing other present data while the automatic backup importer would be done by then.

Added manual export / upload possibility. I don't however see why the backup export should be encrypted in this case. Having it unencrypted on the users hard drive of course would be bad but then, there is no point in encrypting it on the origin pod. The download process could inform user to handle backup storage carefully. However, of course, this means the users private key is in the archive, for the restore path. Do we want to take that route?

If you say the target location is chosen completely automatic, how is the user informed of the backup locations after disaster? It'll be displayed after choosing the backup passphrase and has to be remembered for eternity then?

There are two parts:

  • User is sent an email right after original backup is done by the pod being backed up from
  • User is sent an email right after receiving the first backup on the pod being backed up to

Yes, user would have to not only keep this information in store, but also remember the passphrase. We could do reminder emails every half a year or something. But the passphrase the user must store - that cannot be avoided?

Something that just came to mind, due to the manual procedure. We should probably require the user trying to restore is known to the pod. Otherwise anyone could upload any archive claiming to be someone from a dead pod or even never existed pod by just including a generated key pair. In the same way, we should verify the uploaded private key against the stored profile public key.

@jaywink

Also, btw, of course, if we have a manual archive restore option, that would allow importing posts and conversations. I still wouldn't want this in the automated backups though.

Photos are also tricky, as mentioned in the linked spec. Maybe allow mass importing them from users separate downloaded archive or some kind of "pull" from still active pod as an option. But not in the regular archive imho.

@cmrd-senya

The feature is being crowdfunded. Hope for your support :)

@cmrd-senya

This message doesn't add anything new to what I have posted earlier on other channels (IRC, diaspora, wiki). However, since this is the home page for the issue and it's technical side, this must be referenced here.

Here is my spec for the issue solution. @jaywink's WIP spec was more focused on automatic backup/restore feature, while, I believe this is the last part of the problem. Before implementing automatic backup/restore, we must introduce protocol changes and the restore feature itself. Manual restore doesn't require any other changes, while automatic restore requires some additional features so it's natural to implement manual restore and only after that add the automatic backup/restore capability. Therefore, my spec doesn't have details on the automation, but on all the other things I described.

I have already received the review of the spec from @jhass. I'm waiting for a review by @denschub to move forward with the implementation.

Also, for the record, here are (currently WIP) PRs that are related to this issue resolution: #6726, #6750.

@dcapeletti

Someone knows what the state of development of this feature. 0%, 20%, 50%, 70% 100%

@SuperTux88
diaspora* social network member

see last comment from senya, he is working on it, but there is no exact %-measurement for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment