Support account migration #177

hach-que · Nov 22, 2016

A lot of people seem to be jumping on https://mastodon.social right now, even though the end goal is to have users separated out across multiple federated instances. However, if people start putting up a lot of content and getting followers on the primary instance, this will be a disincentive to move providers.

I think this largely means that Mastodon needs to support two things:

Account import across providers, which should be authenticated on both ends to prevent people bulk copying another person's account, and
A "redirected account" field against user accounts, which indicates the full URL of the new user account

When users configure a redirect location against their account (whether explicitly on an account page, or implicitly set during a cross-provider account import), the instance on which the account is should implicitly and automatically redirect followers.

That is, if I have the account @hq on the primary instance (which I do), and I set up the account @hach-que on another Mastodon service, the @hq should:

Remain on the primary instance, and not be disabled in any way
Show posts from @hach-que after the redirection is set up
Disallow posting from the @hq account while ever the redirection is in place
Existing followers of @hq should start seeing posts from @hach-que instead
New followers of @hq should be allowed to follow the account (and internally, they are following @hq, but see posts from @hach-que)
Followers should not actually have their lists updated to follow @hach-que - in the event of a mistaken redirect, removing the redirect should act exactly as if there never was a redirect in the first place

This is just my 2c on how I think this should work, but I'm interested to hear other people's thoughts.

A "redirected account" field would also be very useful for renaming accounts within a single server.

nex3 · Nov 22, 2016

A "redirected account" field would also be very useful for renaming accounts within a single server.

Sadly there's nothing in the OStatus protocol currently that's like "go follow my new account instead of this one". Best I can offer is migrating content (but reblogs/favs would not come along) and who you were following.

Gargron · Nov 22, 2016

Sadly there's nothing in the OStatus protocol currently that's like "go follow my new account instead of this one". Best I can offer is migrating content (but reblogs/favs would not come along) and who you were following.

I had a bit of a look at the OSocial spec, and it's not 100% clear on what extensions the protocol permits in the <author /> section. All the documentation and links on Portable Contacts (poco) from Wikipedia seem to be dead. Is it possible we can add a new poco tag that contains a redirect or forwarding URL?

edit: or I guess a <link> tag with a different rel attribute?

hach-que · Nov 22, 2016

I had a bit of a look at the OSocial spec, and it's not 100% clear on what extensions the protocol permits in the <author /> section. All the documentation and links on Portable Contacts (poco) from Wikipedia seem to be dead. Is it possible we can add a new poco tag that contains a redirect or forwarding URL?

edit: or I guess a <link> tag with a different rel attribute?

Unless I'm missing something, shouldn't this be protocol-independent? As @hach-que describes it, as far as followers of the original account are concerned, they're still following that account. So as long as the node that that account lives on is able to proxy posts from the new account, this could be managed in a way that's totally transparent to the protocol.

It gets more complicated if you want new followers to automatically follow the new account instead, but that doesn't match the user-visible behavior described above.

nex3 · Nov 22, 2016

Unless I'm missing something, shouldn't this be protocol-independent? As @hach-que describes it, as far as followers of the original account are concerned, they're still following that account. So as long as the node that that account lives on is able to proxy posts from the new account, this could be managed in a way that's totally transparent to the protocol.

It gets more complicated if you want new followers to automatically follow the new account instead, but that doesn't match the user-visible behavior described above.

I've been thinking about this more. It's an important prerequisite for making federation practical, which I know is something @Gargron cares about a lot. We're already seeing a lot of users making accounts and on mastodon.social in particular, and their investment in the community will (hopefully) only grow in the future.

Existing users are the most likely to understand the value of federation, but they're also the most likely to have a lot to lose by abandoning an existing account. At the same time, most people will only move to other servers if the barrier to entry is very low. Allowing accounts to be migrated without losing their investment in the community will remove the biggest part of that barrier.

nex3 · Nov 23, 2016

I've been thinking about this more. It's an important prerequisite for making federation practical, which I know is something @Gargron cares about a lot. We're already seeing a lot of users making accounts and on mastodon.social in particular, and their investment in the community will (hopefully) only grow in the future.

Existing users are the most likely to understand the value of federation, but they're also the most likely to have a lot to lose by abandoning an existing account. At the same time, most people will only move to other servers if the barrier to entry is very low. Allowing accounts to be migrated without losing their investment in the community will remove the biggest part of that barrier.

I wonder if this is something other OStatus implementations would be interested in.

hikari-no-yume · Dec 6, 2016

I wonder if this is something other OStatus implementations would be interested in.

As an in-the-meantime stepping-stone for this, it would be nice if there was a simple import/export of who you're currently following, so that one could easily migrate that over to a new account. It's easy enough to post a “Changing accounts! Follow me at @█████” toot to let followers know you're changing accounts, but significantly harder to ensure you're following the same people on the new one.

marrus-sh · Dec 7, 2016

As an in-the-meantime stepping-stone for this, it would be nice if there was a simple import/export of who you're currently following, so that one could easily migrate that over to a new account. It's easy enough to post a “Changing accounts! Follow me at @█████” toot to let followers know you're changing accounts, but significantly harder to ensure you're following the same people on the new one.

i know that this is an old issue but i'm thinking more so about making a link for accounts, rather than a migration, namely allowing backwards-compatibility, making follow either mean the same account. this seems like it vaguely might be easier than migration? and i think both are very strong and important things to add to making the multiple instance setup for mastodon work

hoodiek · Dec 15, 2016

i know that this is an old issue but i'm thinking more so about making a link for accounts, rather than a migration, namely allowing backwards-compatibility, making follow either mean the same account. this seems like it vaguely might be easier than migration? and i think both are very strong and important things to add to making the multiple instance setup for mastodon work

i would also suggest a profile option at the very least to show where a person has moved. it does not have to automatically move accounts, but making it easier for users to understand that a user has migrated is easy: could make whenever the new account posted, it would remind all followers of the original account to migrate from that they are posting elsewhere and link to the new account, as well as making a single button to unfollow the original and follow the new one.

hoodiek · Dec 15, 2016

i would also suggest a profile option at the very least to show where a person has moved. it does not have to automatically move accounts, but making it easier for users to understand that a user has migrated is easy: could make whenever the new account posted, it would remind all followers of the original account to migrate from that they are posting elsewhere and link to the new account, as well as making a single button to unfollow the original and follow the new one.

if we get REALLY fancy and i would heavily suggest this: the old account could delete the first post to move accounts whenever it posts it again to update that the migrated account has updated a post, to remind users to migrate manually, while also retaining the repository of old posts as a result.

hoodiek · Dec 15, 2016

if we get REALLY fancy and i would heavily suggest this: the old account could delete the first post to move accounts whenever it posts it again to update that the migrated account has updated a post, to remind users to migrate manually, while also retaining the repository of old posts as a result.

Exporting/importing following/blocking lists is great but in my opinion it's not enough. As OP said, people who've tooted will probably stick with their first account instead of moving. I do think that too. I'd like to create an instance and move my account to it but i don't want to get a new account (plus we can't use multiple account at the same time like in TweetDeck). I think an account migration/backup tool is the most missing feature of Mastodon. Complicated sync over instances features are not needed, just a tool to export (CSV or whatever) and delete an account and import it back on another instance would allow people to control their account. Since the instances are managed by the instances' owners, they may be unreliable (reports not processed seriously, 50 % uptime). Then you just have to move.

leeeeeny · Apr 3, 2017

Exporting/importing following/blocking lists is great but in my opinion it's not enough. As OP said, people who've tooted will probably stick with their first account instead of moving. I do think that too. I'd like to create an instance and move my account to it but i don't want to get a new account (plus we can't use multiple account at the same time like in TweetDeck). I think an account migration/backup tool is the most missing feature of Mastodon. Complicated sync over instances features are not needed, just a tool to export (CSV or whatever) and delete an account and import it back on another instance would allow people to control their account. Since the instances are managed by the instances' owners, they may be unreliable (reports not processed seriously, 50 % uptime). Then you just have to move.

Why is it hard to migrate followers? Can't the new system just point to the people on the old system.
Such as if I follow @name on the old system, then when I migrate it then points to @name@mastodon.social or something? Or even just an auto follow and then they have to refollow you?

Maverynthia · Apr 3, 2017

Why is it hard to migrate followers? Can't the new system just point to the people on the old system.
Such as if I follow @name on the old system, then when I migrate it then points to @name@mastodon.social or something? Or even just an auto follow and then they have to refollow you?

The idea is great but the example is a bit complicated, don't you think ?

If I want to migrate my account from mastodon.xyz to another instance, i would want to keep the same username.

Supporting username change should be a "second step" after this first step. (baby steps like neagan says ;))

What I see :
1 - I go the new instance says i want to migrate from mastodon.xyz,
2 - My username is temporary reserved on the new instance
3 - I indicate my login, then it redirects me to mastodon.xyz for authentication (OAuth2)
4 - Once authorized, the migration process begins by importing my parameters in the new instance (following, followers, toots, etc)
5 - delete my account from mastodon.xyz
6 - Enable me on the new instance
7 - update my following with my new address.

my 2cts. (gonna have a lot of cents if we all add up)

gdurelle · Apr 4, 2017

The idea is great but the example is a bit complicated, don't you think ?

If I want to migrate my account from mastodon.xyz to another instance, i would want to keep the same username.

Supporting username change should be a "second step" after this first step. (baby steps like neagan says ;))

What I see :
1 - I go the new instance says i want to migrate from mastodon.xyz,
2 - My username is temporary reserved on the new instance
3 - I indicate my login, then it redirects me to mastodon.xyz for authentication (OAuth2)
4 - Once authorized, the migration process begins by importing my parameters in the new instance (following, followers, toots, etc)
5 - delete my account from mastodon.xyz
6 - Enable me on the new instance
7 - update my following with my new address.

my 2cts. (gonna have a lot of cents if we all add up)

Yes that's a bit complicated but imo this is a very important feature to permit existing users balancing between the instances and avoid account loss in case of instance shutdown.

leeeeeny · Apr 4, 2017

Yes that's a bit complicated but imo this is a very important feature to permit existing users balancing between the instances and avoid account loss in case of instance shutdown.

I agree with everyone here. Account migration is necessary. The most important is toots and followers for me. Followings too but they can already be exported. Username would be a good thing too.

Sadzeih · Apr 4, 2017

I agree with everyone here. Account migration is necessary. The most important is toots and followers for me. Followings too but they can already be exported. Username would be a good thing too.

Account migration is a necessity if you want federation to be a first-class citizen. Users need to be insulated from the risks associated with decentralization. If an instance goes away (crashes, is hacked, shut down by an oppressive regime) those users should not be SOL. Lacking that, users will flock to the instance with the highest likelihood of surviving.

Motoma · Apr 4, 2017

Account migration is a necessity if you want federation to be a first-class citizen. Users need to be insulated from the risks associated with decentralization. If an instance goes away (crashes, is hacked, shut down by an oppressive regime) those users should not be SOL. Lacking that, users will flock to the instance with the highest likelihood of surviving.

Well, there's two ways (in my mind) of fixing the problem:

You do it the GitHub way, where the old username is now forwarded to the new username. I'm not sure if OStatus has a concept of 301 redirects, but that's how you would do it. The downside is that if the server goes down before a client has got the redirect they're SoL.
You handle it the Matrix way, which is that you have identity servers that decouple the "real" ID (@username@instance) from the person whose ID it belongs to. The downside is that Matrix implemented this in a very centralised way, which just punts the issue. Not to mention OStatus almost certainly doesn't have support for such a concept.

cyphar · Apr 4, 2017

Well, there's two ways (in my mind) of fixing the problem:

You do it the GitHub way, where the old username is now forwarded to the new username. I'm not sure if OStatus has a concept of 301 redirects, but that's how you would do it. The downside is that if the server goes down before a client has got the redirect they're SoL.
You handle it the Matrix way, which is that you have identity servers that decouple the "real" ID (@username@instance) from the person whose ID it belongs to. The downside is that Matrix implemented this in a very centralised way, which just punts the issue. Not to mention OStatus almost certainly doesn't have support for such a concept.

@TehShrike This issue is about moving between Mastodon instances, not from Twitter to Mastodon.

cyphar · Apr 4, 2017

@TehShrike This issue is about moving between Mastodon instances, not from Twitter to Mastodon.

Ah, I googled-and-skimmed too quickly >_< I'll delete that post.

TehShrike · Apr 4, 2017

Ah, I googled-and-skimmed too quickly >_< I'll delete that post.

My hand-wave-y two cents:

It would be nice if there was a decoupled notion of identity. For example: a keybase.io profile aggregates different accounts for a single person so that they can be easily identified across all networks.

If this were true, you could imagine a system where you follow someone on Mastodon by following an identity and all addresses (that is, instance/profile combinations) that are associated with it.

Of course, it isn't easy to maintain a trusted database of identities that is shareable across all federated nodes. But, it's possible. Perhaps we can discuss the feasibility of such an approach.

👋

cdata · Apr 4, 2017

My hand-wave-y two cents:

It would be nice if there was a decoupled notion of identity. For example: a keybase.io profile aggregates different accounts for a single person so that they can be easily identified across all networks.

If this were true, you could imagine a system where you follow someone on Mastodon by following an identity and all addresses (that is, instance/profile combinations) that are associated with it.

Of course, it isn't easy to maintain a trusted database of identities that is shareable across all federated nodes. But, it's possible. Perhaps we can discuss the feasibility of such an approach.

👋

Blockstack might be a good candidate for providing a decentralised, global identity. With Blockstack you could keep the same ID (and thus your followers) even if the instance you are currently using goes down permanently.

stiell · Apr 4, 2017

Blockstack might be a good candidate for providing a decentralised, global identity. With Blockstack you could keep the same ID (and thus your followers) even if the instance you are currently using goes down permanently.

The problem with having separated identities is that it wouldn't be compatible with other OStatus implementations (Mastodon isn't the only implementation, GNU Social is another and there's quite a few more). Migration should be something that someone following a Mastodon user from GNU Social should be able to support.

cyphar · Apr 4, 2017

The problem with having separated identities is that it wouldn't be compatible with other OStatus implementations (Mastodon isn't the only implementation, GNU Social is another and there's quite a few more). Migration should be something that someone following a Mastodon user from GNU Social should be able to support.

In the fullness of time, I agree that this should be something that is broadly compatible with GNU Social. In the isolation of the moment, I propose that evolution requires experimentation. Compatibility is a matter of adding sufficient layers to make the standardized things cooperate with the experimental thing.

My brain told my hand to stop waving. I'm sure it will stop soon.

cdata · Apr 4, 2017

In the fullness of time, I agree that this should be something that is broadly compatible with GNU Social. In the isolation of the moment, I propose that evolution requires experimentation. Compatibility is a matter of adding sufficient layers to make the standardized things cooperate with the experimental thing.

My brain told my hand to stop waving. I'm sure it will stop soon.

I think you guys are making a mountain out of a molehill here. I think this problem is pretty easy to solve. In order of least to most controversial: we should make it possible to import and export toots, we should make an automated migration tool (log into your old account via OAuth from your new instance), and we should toot out a notice of account change.

The migration toot would be a normal toot with a human-readable message ("My account is moving to (link)"), but would include something like this:

<mastodon:migrate participant="origin">https://new-account</mastodon:migrate>

The new account would toot something like this to confirm it ("Moved from my old account (link)"):

<mastodon:migrate participant="destination">https://old-account</mastodon:migrate>

Then, clients can recognize this attribute and perform the migration when they see the toot by unfollowing and following the new account (possibly with an another XML extension that indicates the user is responding to an account migration). Clients would likely want to hide the special toots from the feed as well. Clients that don't support it have the human-readable version as a fallback. Then we just send patches to GNU Social and other projects to recognize this behavior.

SirCmpwn · Apr 4, 2017

I think you guys are making a mountain out of a molehill here. I think this problem is pretty easy to solve. In order of least to most controversial: we should make it possible to import and export toots, we should make an automated migration tool (log into your old account via OAuth from your new instance), and we should toot out a notice of account change.

The migration toot would be a normal toot with a human-readable message ("My account is moving to (link)"), but would include something like this:

<mastodon:migrate participant="origin">https://new-account</mastodon:migrate>

The new account would toot something like this to confirm it ("Moved from my old account (link)"):

<mastodon:migrate participant="destination">https://old-account</mastodon:migrate>

Then, clients can recognize this attribute and perform the migration when they see the toot by unfollowing and following the new account (possibly with an another XML extension that indicates the user is responding to an account migration). Clients would likely want to hide the special toots from the feed as well. Clients that don't support it have the human-readable version as a fallback. Then we just send patches to GNU Social and other projects to recognize this behavior.

I'll probably start working on a toot export/import tool shortly, in any case.

SirCmpwn · Apr 4, 2017

I'll probably start working on a toot export/import tool shortly, in any case.

@TehShrike There IS this tool for what you are looking for: https://mastodon-bridge.herokuapp.com/ but that's only if they signed up on both.

Maverynthia · Apr 4, 2017

@TehShrike There IS this tool for what you are looking for: https://mastodon-bridge.herokuapp.com/ but that's only if they signed up on both.

The way to connect accounts is XFN - rel information on links. You use rel=me to say "this is also me" and if both urls do that to each other it's confirmed.
rel="friend" will do for following (there are others, but that is simple enough to use).

If you want to convey "I moved there" use rel="canonical me"

The added benefit of this is that you can do actual distributed verification tags - confirming people's twitter/github/medium/whatever accounts too.

kevinmarks · Apr 5, 2017

The way to connect accounts is XFN - rel information on links. You use rel=me to say "this is also me" and if both urls do that to each other it's confirmed.
rel="friend" will do for following (there are others, but that is simple enough to use).

If you want to convey "I moved there" use rel="canonical me"

The added benefit of this is that you can do actual distributed verification tags - confirming people's twitter/github/medium/whatever accounts too.

I'm not sure those fit ideally into this problem, @kevinmarks. They're for duplicated content, not a one-time notification that the content publisher has moved. Would make more sense if we were mirroring toots between multiple accounts but I think that's unrelated to the matter at hand.

SirCmpwn · Apr 5, 2017

I'm not sure those fit ideally into this problem, @kevinmarks. They're for duplicated content, not a one-time notification that the content publisher has moved. Would make more sense if we were mirroring toots between multiple accounts but I think that's unrelated to the matter at hand.

the point about rel's is that they are amenable to setting up bidirectional verification of the move. Then you can use a 301 redirect for moved URLs.

kevinmarks · Apr 5, 2017

the point about rel's is that they are amenable to setting up bidirectional verification of the move. Then you can use a 301 redirect for moved URLs.

I think I see what you mean. The one-time "I've moved" toots could use rel="me" (new account) and rel="canonical me" (old account) to indicate the move? I'm still not sure that's entirely in the spirit of these attributes but I don't see why it wouldn't work.

SirCmpwn · Apr 5, 2017

I think I see what you mean. The one-time "I've moved" toots could use rel="me" (new account) and rel="canonical me" (old account) to indicate the move? I'm still not sure that's entirely in the spirit of these attributes but I don't see why it wouldn't work.

No, they shouldn't be toots, they should be in the bio section

…

On 5 Apr 2017 1:31 am, "Drew DeVault" ***@***.***> wrote: I think I see what you mean. The one-time "I've moved" toots could use rel="me" (new account) and rel="canonical me" (old account) to indicate the move? I'm still not sure that's entirely in the spirit of these attributes but I don't see why it wouldn't work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#177 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGCwJHPYajcxkpXohajzF5z9LASspUdks5rsuDxgaJpZM4K5Bba> .

kevinmarks · Apr 5, 2017

No, they shouldn't be toots, they should be in the bio section

…

On 5 Apr 2017 1:31 am, "Drew DeVault" ***@***.***> wrote: I think I see what you mean. The one-time "I've moved" toots could use rel="me" (new account) and rel="canonical me" (old account) to indicate the move? I'm still not sure that's entirely in the spirit of these attributes but I don't see why it wouldn't work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#177 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGCwJHPYajcxkpXohajzF5z9LASspUdks5rsuDxgaJpZM4K5Bba> .

Aha, good idea. Makes sense.

SirCmpwn · Apr 5, 2017

Aha, good idea. Makes sense.

Doesn't importing a list of exported toots represent a pretty major potential vulnerability with regard to spamming? For instance, somebody could make a large synthetic export file with faked timestamps and flood the local timeline on a new account, or create new one-off follower accounts on several major nodes first and flood several federated timelines, at a scale larger than manual posting, if my understanding of how federation works is correct.

Don't get me wrong -- I think transferring full toot history is important for account migration, and I think we should have an export function for toots available to all users -- but I think full account transfer should probably be negotiated between federated servers (even if it's a separate protocol or an extension), and that toot history import should either: 1) be heavily limited in scale (say, no more than 100); 2) be omitted from inclusion in the local and federated timelines; or 3) have retroactively-applied rate limits at least as strict as the ones applied to people posting through the existing front end, so that any file with toot timestamps too close together gets rejected.

enkiv2 · Apr 5, 2017

Doesn't importing a list of exported toots represent a pretty major potential vulnerability with regard to spamming? For instance, somebody could make a large synthetic export file with faked timestamps and flood the local timeline on a new account, or create new one-off follower accounts on several major nodes first and flood several federated timelines, at a scale larger than manual posting, if my understanding of how federation works is correct.

Don't get me wrong -- I think transferring full toot history is important for account migration, and I think we should have an export function for toots available to all users -- but I think full account transfer should probably be negotiated between federated servers (even if it's a separate protocol or an extension), and that toot history import should either: 1) be heavily limited in scale (say, no more than 100); 2) be omitted from inclusion in the local and federated timelines; or 3) have retroactively-applied rate limits at least as strict as the ones applied to people posting through the existing front end, so that any file with toot timestamps too close together gets rejected.

A user opinion about the priority when migrating: make followers auto-follow the new account.

I believe that mastodon, as other microblogging platforms, is mainly focused on real-time.

I can deal with going back to a clean toots history but starting again from 0 followers would totally discourage me to migrate to another instance. And I know that a "I moved here re-follow me!" notification won't be enough to bring back half of the people that were in my network.

What is really precious in an account is its followers base.

thomaslule · Apr 5, 2017

A user opinion about the priority when migrating: make followers auto-follow the new account.

I believe that mastodon, as other microblogging platforms, is mainly focused on real-time.

I can deal with going back to a clean toots history but starting again from 0 followers would totally discourage me to migrate to another instance. And I know that a "I moved here re-follow me!" notification won't be enough to bring back half of the people that were in my network.

What is really precious in an account is its followers base.

Actually, I don't think migration where the original instance has shut down impossible. Maybe not even that hard. I think the key concept that makes it feasible is having a backup account at another instance, set up in advance. Like with computer data backups, this should be a thing that people are encourage to do if they're using the system for anything serious, and hopefully it should be pretty easy to do.

So, my primary account is @sandro@w3c.social, and I'd probably make my backup be my older account, @sandhawke@mastodon.social. There should be a way for me to point the accounts at each other, so every subscribing system has recorded that one is the backup for the other, and the servers should be set up do be replicating my data.

If the primary server goes away, even suddenly, all the followers can automatically use the backup. If the backup goes away suddenly, then I pick a new backup. The only catastrophic failure would be if both went away at about the same time. Maybe the system could even support multiple backup servers at once, for the truly paranoid.

sandhawke · Apr 19, 2018

Actually, I don't think migration where the original instance has shut down impossible. Maybe not even that hard. I think the key concept that makes it feasible is having a backup account at another instance, set up in advance. Like with computer data backups, this should be a thing that people are encourage to do if they're using the system for anything serious, and hopefully it should be pretty easy to do.

So, my primary account is @sandro@w3c.social, and I'd probably make my backup be my older account, @sandhawke@mastodon.social. There should be a way for me to point the accounts at each other, so every subscribing system has recorded that one is the backup for the other, and the servers should be set up do be replicating my data.

If the primary server goes away, even suddenly, all the followers can automatically use the backup. If the backup goes away suddenly, then I pick a new backup. The only catastrophic failure would be if both went away at about the same time. Maybe the system could even support multiple backup servers at once, for the truly paranoid.

@pingu8007:

I just had a look at this thread, the issue can split into:
Data transfer - export and import
User discovery - finding moved user

Data transfer is the biggest holdout right now; user discovery is currently handled by redirection notices (better than nothing, but can be improved). To try and lay it all out again as hq did (slightly modified for easier referencing):

That is, if I have the account A on the primary instance (which I do), and I set up the account B on another Mastodon service, then A should:

1) Remain on the primary instance, and not be disabled in any way
2) Show posts from B after the redirection is set up
3) Disallow posting from the A account while ever the redirection is in place
4) Existing followers of A should start seeing posts from @hach-que instead
5) New followers of A should be allowed to follow the account (and internally, they are following A, but see posts from B)
6) Followers should not actually have their lists updated to follow B - in the event of a mistaken redirect, removing the redirect should act exactly as if there never was a redirect in the first place

And right now, we have the following:

1) A remains on the original instance, albeit with a banner indicating the new account
2) A does not show posts from B, but only the old posts from A
3) Redirection doesn't disallow posting from A; people post on A to tell people to follow B
4) Followers of A do not see posts from B -- they must refollow B
5) A is greyed out and idk if following is fully disabled; I haven't really looked at it recently
6) Followers must refollow B instead of automatically being forced to follow B.

trwnh · Apr 19, 2018

@pingu8007:

I just had a look at this thread, the issue can split into:
Data transfer - export and import
User discovery - finding moved user

Data transfer is the biggest holdout right now; user discovery is currently handled by redirection notices (better than nothing, but can be improved). To try and lay it all out again as hq did (slightly modified for easier referencing):

That is, if I have the account A on the primary instance (which I do), and I set up the account B on another Mastodon service, then A should:

1) Remain on the primary instance, and not be disabled in any way
2) Show posts from B after the redirection is set up
3) Disallow posting from the A account while ever the redirection is in place
4) Existing followers of A should start seeing posts from @hach-que instead
5) New followers of A should be allowed to follow the account (and internally, they are following A, but see posts from B)
6) Followers should not actually have their lists updated to follow B - in the event of a mistaken redirect, removing the redirect should act exactly as if there never was a redirect in the first place

And right now, we have the following:

1) A remains on the original instance, albeit with a banner indicating the new account
2) A does not show posts from B, but only the old posts from A
3) Redirection doesn't disallow posting from A; people post on A to tell people to follow B
4) Followers of A do not see posts from B -- they must refollow B
5) A is greyed out and idk if following is fully disabled; I haven't really looked at it recently
6) Followers must refollow B instead of automatically being forced to follow B.

@sandhawke You described "nomadic identity" in the zot protocol / hubzilla that @trwnh pointed to a few messages earlier (Jan 1st). However, it's not all clear to me. It seems like you end up in a master-replica situation with two masters...

Gargron · Apr 19, 2018

@sandhawke You described "nomadic identity" in the zot protocol / hubzilla that @trwnh pointed to a few messages earlier (Jan 1st). However, it's not all clear to me. It seems like you end up in a master-replica situation with two masters...

@Gargron It could very confusing, yes. How about this, instead -- some accounts are explicit backup accounts. I'm sandro@w3c.social and backup_for_sandro_w3c_social@mastodon.social (more or less). The backup accounts can't be posted to; they only get content via sync from their master. The only thing the user can do at the backup server is point it to a new master if the old master dies.

sandhawke · Apr 19, 2018

@Gargron It could very confusing, yes. How about this, instead -- some accounts are explicit backup accounts. I'm sandro@w3c.social and backup_for_sandro_w3c_social@mastodon.social (more or less). The backup accounts can't be posted to; they only get content via sync from their master. The only thing the user can do at the backup server is point it to a new master if the old master dies.

@sandhawke You're correct that migration from a shut-down instance isn't impossible, per se... it just requires you to proactively download your exported archive. If the instance goes down suddenly like mastodon.rocks, then you likely didn't make a backup, and now all the data is gone. Under such a system, the onus would be on every single user to make regular backups if they care about migrating off the instance or in the event of catastrophic failure. This is a very awkward solution.

So, my primary account is @sandro@w3c.social, and I'd probably make my backup be my older account, @sandhawke@mastodon.social. There should be a way for me to point the accounts at each other, so every subscriber knows one is the backup for the other, and the servers should be set up do be replicating my data.

What you're describing is the live migration / mirroring, or "nomadic identity" system that Hubzilla uses: #177 (comment)

Under zot, any linked accounts are effectively mirrors of each other, passed back and forth between servers. Any server that is hosting your identity will share basically everything back and forth -- mentions, permissions, post history, etc. And yes, if one goes down, then everything is fine as long as any other clone is up.

@Gargron I'm willing to attempt to do what it takes to clarify the situation or implementation in any way. I suppose "master-replica with two masters" is pretty close, yes -- it's like sending an email message to two addresses, and therefore you end up with two copies in two inboxes. But each copy is still the same message; you just have two copies of it.

Conceptually (http://www.talkplus.org/blog/2017/what-is-a-nomadic-identity/):

Bob has two email accounts. He has a home account at bob@home.server, and he has a work account at bob@work.server [...] when we send an email, we will send to both of Bob’s addresses any time we send an email to Bob. This way he’ll get the message no matter if he’s at home or if he’s at work. [...] as far as your software is concerned, it’s just ‘Bob’. You don’t care what server he uses or what job he has this week. [...] if Bob loses his job and his account gets removed from work.server [...] he’ll just use home.server; and then when he gets a new job, he can send you stuff from bob@newjob.server. [...] if Bob posts a picture while he’s at work, his work server sends a copy to his home server so he’ll have the same picture in his photo albums in both places. If he makes a new friend, the friend will be added on both servers so he has the same friends no matter where he goes.

Technically (https://project.hubzilla.org/help/en/developer/zot_protocol#Technical_Introduction):

In order for an identity to persist across locations, one must be able to provide or recover:

the globally unique ID for that identity

the private key assigned to that identity

(if the original server no longer exists) an address book of contacts for that identity.
This information will be exportable from the original server via API, and/or downloadable to disk or thumb-drive.

Generating GUID (globally unique ID):

to create a globally unique ID, we will base it on a whirlpool hash of the identity URL of the origination node and a psuedo-random number, which should provide us with a 256 bit ID with an extremely low probability of collision, as a base64url-encoded string. [...] the ID must also be attached to a public key [...] We will refer to the encoded globally unique uid string as*zot_uid

With regards to DNS and a "primary" hub:

As there may be more than one DNS location attached/bound to a given zot_uid identity, delivery processes should deliver to all of them - as we do not know for sure which hub instance may be accessed at any given time. However we will designate one DNS location as "primary" and which will be the preferred location to view web profile information. We must also provide the ability to change the primary to a new location

How the primary hub/node works:

A look-up of information on the current primary location may provide a "forwarding pointer" which will tell us to update our records and move everything to the new location. There is also the possibility that the primary location is defunct and no longer responding. In that case, a location which has already been mapped to this zot_uid can seize control, and declare itself to be primary. In both cases the primary designation is automatically approved and moved. A request can also be made from a primary location which requests that another location be removed.

How to map additional hubs:

In order to map a zot_uid to a second (or tertiary) location, we require a secure exchange which verifies that the new location is in possession of the private key for this zot_uid. Security of the private key is thus essential to avoid hijacking of identities. [...] Key revocation and replacement must take place from the primary hub.

How account discovery occurs:

A well-known url is used to probe a hub for Zot capabilities and identity lookups, including the discovery of public keys, profile locations, profile photos, and primary hub location. The location for this service is /.well-known/zot-info - and must be available at the root of the chosen domain.

I advise looking at the example discovery packet to get a better idea of its structure, but tldr: it contains a signed token, signed guid, public key, some other profile metadata, and importantly, arrays for locations and for the site you're communicating with. Specifically, the locations array contains a hostname, address, whether it is primary, a signed url, a callback, and a site public key, for each location that is hosting the zot_uid.

tl;dr for all that: Hubzilla / zot works by creating guids and signing with some sort of PKI, and if your primary server authorizes any other servers, then those servers get deliveries of the same messages.

The guid is the hard part. Once you have guids (or, in Mastodon's case, once existing accounts are somehow converted to / assigned new guids), you can now address messages to guids instead of to local uid or username@domain. And like I said on #6837: the sooner this is done, the less headaches it will cause in the future. Trying to refactor something that affects a million accounts is easier than trying to refactor something that affects ten million, or a hundred million. As long as there's a dependence on DNS identity, any solution to account migration can't ever be 100% seamless. Perhaps that's a tradeoff willingly made, or perhaps not.

trwnh · Apr 19, 2018

@sandhawke You're correct that migration from a shut-down instance isn't impossible, per se... it just requires you to proactively download your exported archive. If the instance goes down suddenly like mastodon.rocks, then you likely didn't make a backup, and now all the data is gone. Under such a system, the onus would be on every single user to make regular backups if they care about migrating off the instance or in the event of catastrophic failure. This is a very awkward solution.

So, my primary account is @sandro@w3c.social, and I'd probably make my backup be my older account, @sandhawke@mastodon.social. There should be a way for me to point the accounts at each other, so every subscriber knows one is the backup for the other, and the servers should be set up do be replicating my data.

What you're describing is the live migration / mirroring, or "nomadic identity" system that Hubzilla uses: #177 (comment)

Under zot, any linked accounts are effectively mirrors of each other, passed back and forth between servers. Any server that is hosting your identity will share basically everything back and forth -- mentions, permissions, post history, etc. And yes, if one goes down, then everything is fine as long as any other clone is up.

@Gargron I'm willing to attempt to do what it takes to clarify the situation or implementation in any way. I suppose "master-replica with two masters" is pretty close, yes -- it's like sending an email message to two addresses, and therefore you end up with two copies in two inboxes. But each copy is still the same message; you just have two copies of it.

Conceptually (http://www.talkplus.org/blog/2017/what-is-a-nomadic-identity/):

Bob has two email accounts. He has a home account at bob@home.server, and he has a work account at bob@work.server [...] when we send an email, we will send to both of Bob’s addresses any time we send an email to Bob. This way he’ll get the message no matter if he’s at home or if he’s at work. [...] as far as your software is concerned, it’s just ‘Bob’. You don’t care what server he uses or what job he has this week. [...] if Bob loses his job and his account gets removed from work.server [...] he’ll just use home.server; and then when he gets a new job, he can send you stuff from bob@newjob.server. [...] if Bob posts a picture while he’s at work, his work server sends a copy to his home server so he’ll have the same picture in his photo albums in both places. If he makes a new friend, the friend will be added on both servers so he has the same friends no matter where he goes.

Technically (https://project.hubzilla.org/help/en/developer/zot_protocol#Technical_Introduction):

In order for an identity to persist across locations, one must be able to provide or recover:

the globally unique ID for that identity

the private key assigned to that identity

(if the original server no longer exists) an address book of contacts for that identity.
This information will be exportable from the original server via API, and/or downloadable to disk or thumb-drive.

Generating GUID (globally unique ID):

to create a globally unique ID, we will base it on a whirlpool hash of the identity URL of the origination node and a psuedo-random number, which should provide us with a 256 bit ID with an extremely low probability of collision, as a base64url-encoded string. [...] the ID must also be attached to a public key [...] We will refer to the encoded globally unique uid string as*zot_uid

With regards to DNS and a "primary" hub:

As there may be more than one DNS location attached/bound to a given zot_uid identity, delivery processes should deliver to all of them - as we do not know for sure which hub instance may be accessed at any given time. However we will designate one DNS location as "primary" and which will be the preferred location to view web profile information. We must also provide the ability to change the primary to a new location

How the primary hub/node works:

A look-up of information on the current primary location may provide a "forwarding pointer" which will tell us to update our records and move everything to the new location. There is also the possibility that the primary location is defunct and no longer responding. In that case, a location which has already been mapped to this zot_uid can seize control, and declare itself to be primary. In both cases the primary designation is automatically approved and moved. A request can also be made from a primary location which requests that another location be removed.

How to map additional hubs:

In order to map a zot_uid to a second (or tertiary) location, we require a secure exchange which verifies that the new location is in possession of the private key for this zot_uid. Security of the private key is thus essential to avoid hijacking of identities. [...] Key revocation and replacement must take place from the primary hub.

How account discovery occurs:

A well-known url is used to probe a hub for Zot capabilities and identity lookups, including the discovery of public keys, profile locations, profile photos, and primary hub location. The location for this service is /.well-known/zot-info - and must be available at the root of the chosen domain.

I advise looking at the example discovery packet to get a better idea of its structure, but tldr: it contains a signed token, signed guid, public key, some other profile metadata, and importantly, arrays for locations and for the site you're communicating with. Specifically, the locations array contains a hostname, address, whether it is primary, a signed url, a callback, and a site public key, for each location that is hosting the zot_uid.

tl;dr for all that: Hubzilla / zot works by creating guids and signing with some sort of PKI, and if your primary server authorizes any other servers, then those servers get deliveries of the same messages.

The guid is the hard part. Once you have guids (or, in Mastodon's case, once existing accounts are somehow converted to / assigned new guids), you can now address messages to guids instead of to local uid or username@domain. And like I said on #6837: the sooner this is done, the less headaches it will cause in the future. Trying to refactor something that affects a million accounts is easier than trying to refactor something that affects ten million, or a hundred million. As long as there's a dependence on DNS identity, any solution to account migration can't ever be 100% seamless. Perhaps that's a tradeoff willingly made, or perhaps not.

Two observations on the current conversation:

If a solution similar to nomadic identity is to be implemented, it would be best for it to be fully automated and for a user to be able to choose to migrate to any instance, preserving at least their follow/following lists, at any time, without doing any work ahead of time. If migration requires planning and work up front, most people aren't going to do it. New users still have trouble with the concept of instances. If they even realize that migration is a concern when signing up, many of them would hit the (inevitably complicated) section of docs about backup accounts and migration and say "wat" (usually) or "k I'll do this later" (if we're lucky). Then most of them won't set it up, and when instances vanish, they're screwed, even though all this dev work was spent on a migration feature. Don't get me wrong, it's great to support the garden path of migrating between running instances, but can we do better? (See below for implementation details.)
If a user chooses to migrate, they really should be informed when the destination instance has conflicts with their follow/following lists due to instance blocks - this needs to be addressed from the start and not when somebody inevitably files a bug after they do all the work to migrate to a new instance and only then find out their favorite follows/followers are blocked.

Implementation details for zero-config migration:

You'd need a globally unique identifier held only by the user. Is the tuple (username, password_hash, optional_2fa_seed) sufficient to authenticate the user and pair them with e.g. an immutable uuid used behind the scenes?

Information sufficient to authenticate the user and to reconstitute their follow/following lists would need to be replicated around the network. Storing it all in a blockchain is overkill - not every node should need to retain every user's follow lists, that doesn't scale. However, the tuple used to authenticate users might need to be more widely replicated than follow lists.

To keep network and storage load sane, you could store follow lists on some number of instances n considered sufficient to provide redundancy (say n=3) but then you have the problem of which instances store what, how do you retrieve the info when needed, and how do you distribute updates when follow lists or passwords change.

Focusing on just the follow lists for the moment, one could store them in a DHT with the key being the global uuid of the user, and have some sort of election to pick the n instances to store the data at the time the user account is first created. The election process could be as simple as "build me a hash table of all active instances, deterministically select n based on the value of the user's uuid" or something more equitable (and potentially controllable) for server operators, like a population-weighted selection algorithm, so small instances don't end up hosting tons of data disproportionate to their number of users.

Once you have that infrastructure, you need a couple more things.

When one of the n instances replicating the follow lists goes away for long enough (i.e. availability on the DHT drops below n for too long), some process needs to notice this and elect a new instance to host replication data such that DHT availability is brought back up to n. Likely this should be handled by the user's home instance - that seems fair in terms of load. Eventual consistency is fine here, this process can operate at a slow and steady pace, and only get worried when it notices availability consistently < n for some value sufficient to account for outages likely to be seen on Mastodon (perhaps 3 days).

Perhaps more difficult, when the user's follow/following lists change, updates need to be replicated. Again, eventual consistency is fine here. Allowing it to take a day or so for updates to replicate is probably fine, especially if it balances load. But more importantly: how do you prevent spoofing if the user's uuid and the instances hosting their replicated data are known? And does this change the minimum necessary tuple of (username, password_hash, optional_2fa_seed) necessary to authenticate the user, or processes operating on their behalf?

The way I see it, the major costs of this scheme are development cost - lines of code to write, debug, and maintain - and server load - mostly additional disk space for storing user auth tuples and follow/following lists, and network traffic replicating updates to follow/following lists. It may make sense to consider the idea of supernodes at this point, or some other mechanism to make the scheme opt-in for small instances, perhaps as simple as until you have 100 monthly active users, hosting replication data is opt-in.

The major benefits, however, may be worth all this work. It really depends on how much the feature is likely to be used. If people would tend to use it often enough, doing it this way would make mastodon exceptionally resilient to the loss of instances, even huge ones, without users having to do any work up front at all. A big lesson one can take from natural disasters is that when a city gets devastated, people whose housing is destroyed often move somewhere else and don't return. What happens when a top 10 instance vanishes without warning? I think the answer is a lot different if the users can simply log in at any other instance and choose to migrate their presence there, even with the original instance offline.

Edit: there are security and privacy concerns with parts of my scheme, particularly with replication of user auth data - I suspect somebody more studied than me in CS and/or crypto can come up with something similar but better. With regard to replicating follow lists, this is a privacy concern, but I'd argue that having some number of instances n with a user's follow lists is an acceptable trade-off, given that there is not, and will never be, any guarantee that the data cannot be assembled by other means conceptually far simpler and more likely to work than attempting to maliciously host replicated follow lists - see #6901.

deutrino · Apr 19, 2018

Two observations on the current conversation:

If a solution similar to nomadic identity is to be implemented, it would be best for it to be fully automated and for a user to be able to choose to migrate to any instance, preserving at least their follow/following lists, at any time, without doing any work ahead of time. If migration requires planning and work up front, most people aren't going to do it. New users still have trouble with the concept of instances. If they even realize that migration is a concern when signing up, many of them would hit the (inevitably complicated) section of docs about backup accounts and migration and say "wat" (usually) or "k I'll do this later" (if we're lucky). Then most of them won't set it up, and when instances vanish, they're screwed, even though all this dev work was spent on a migration feature. Don't get me wrong, it's great to support the garden path of migrating between running instances, but can we do better? (See below for implementation details.)
If a user chooses to migrate, they really should be informed when the destination instance has conflicts with their follow/following lists due to instance blocks - this needs to be addressed from the start and not when somebody inevitably files a bug after they do all the work to migrate to a new instance and only then find out their favorite follows/followers are blocked.