Support Post Migration #12423
Comments
|
Once more I fully support this and have been eagerly awaiting the much requested feature. I would love to see the ability to import the data archives (favorites boosts and toots) exported from the https://instance.social/settings/export section, which is currently a one-way system only. Unfortunately there doesn't seem to be a plan for this yet, but I'm staying hopeful that will change eventually. I believe Eugen mentioned performance concerns on the previous discussion. |
|
Reiterating points from previous discussion:
|
|
Thanks @trwnh!
What's backfilling? |
|
backfilling is #34 -- fetching old statuses from a profile that were never delivered to the instance. you can't expect to deliver 100,000 activities every time you migrate, because that's wildly inefficient. |
|
Regarding performance and backfilling: This is why I believe that post migration is possible, but those who choose to use it will need to accept a long waiting period. There will probably have to be three types of limitations to make it viable:
Obviously, if you're importing something like 10.000 boosts + faves + toots, the process may take days if not weeks. It's a price the user will have to pay. But in this format it sounds like it should be performance viable at least. |
|
That makes sense! Personally I'm perfectly fine with them taking a long while to move over, so long as they move over eventually. I'm not quite sure what the limit should be, though – one post per 10 seconds seems awfully slow to me, unless you have like 100 accounts being migrated at once. Maybe set it relatively fast by default (like 1/s or maybe even a little higher), since most instances are small and won't need to deal with a lot of migrations, then larger instances can set it slower? |
|
Although I would really, really prefer if everything gets migrated, even if it takes weeks or even months. I don't want to lose my old posts – that's basically an archive of what I'm like, so new people can get to know me a little better before they decide whether to follow me. It's also history, and while some may not want to keep that sort of thing, others (like me) would. (Even if you technically have everything in the backup, that's incredibly cumbersome to look at, given I'm not aware of any software that can actually import it.) |
|
stuff can get imported, but it is absolutely a bad idea to redeliver it. i am firmly of the opinion that a Move with post import should maybe do a rewrite of existing statuses in the database, but nothing more. it is unwise to even consider backfilling until the actual decoupling and importing gets implemented. more to the point in general, though: it is fundamentally bad idea to assume that every instance should have a copy of every single post at all times. we don't need to duplicate everything 6000x or more. what is fundamentally happening with a "migration" is that the authority is being shifted; the location is being shifted, that's all. it's a Bad Idea to treat them as new statuses. hopefully i've explained why |
|
Hm...well, would they be new statuses? I figured they would simply be reassigned, but then maybe redelivered as the originals, not duplicated and "reposted". Or maybe I'm completely misunderstanding how federation works. I know very little about the backend.
Is this actually being discussed? I'd say for a migration the situation is a bit different; only the destination server has to receive a copy of every single post, not everybody. |
yeah, and it should be limited to this. looking at prior art from zot, you can do an online migration (fetch old messages from your outbox and inbox to the new server) or an offline migration (using an exported account archive that contains at minimum your keys, your posts, and your address book) |
|
That sounds lovely! I've been a little worried in the back of my mind about how to handle a server that just suddenly dies. |
|
Is there at least a first-party tool that users (not servers) can use to work their downloaded backup and generate something usable? Something like a HTML file (plus media?) that can be uploaded into a remote server. Would at least temporarily solve the problem of making the lost content accessible again. |
|
@lmachucab I made a NodeJS script that I used to port my faves and boosts to other accounts. Unfortunately it no longer worked last time I tried it, and it was a crude solution that put quite a bit of stress on the target server and even triggered an instance's flood protections once. If you think you might have some use for it still, it's still available on my Github: |
|
Hi there! Mastodon.cloud literally has a month to live. We're being shut down because the domain is being forced to bow to insane regulations, and the admin isn't having any of it. We had been the target of immense DDoS attacks for the past six months. |
|
I'd like to second that Mastodon.cloud disappearing shows this is a crucial feature. Mastodon is a great piece of software, and designing it to be decentralized was a great idea. Making it possible for anyone to set up an instance, yet having instances be able to communicate with one another, makes the network resilient and eliminates single points of failure. But without the ability to jump from one instance to another at will, this decentralization has a large drawback. If an instance decides to call it quits, you're just going to lose your data. This also means that as a user, you can't be sure if the instance you picked is going to live a long or a short life. Given how many instances there are, it's totally reasonable to expect that some of them are going to disappear. Aside from users choosing to migrate somewhere else, there should also be a migration plan for instance admins. If an instance shuts down, it shouldn't just be that active users who care about their account get to keep their data, because that means even if user migration is possible there's still going to be a big loss of online culture and data. I think this should probably be a separate issue and come with a host of problems by itself, but without it I think we'll see Mastodon content regularly disappearing forever as instances come and go. |
|
Yeah even I didn't expect this sad news. mastodon.cloud banned me randomly and without explanation after a few months of using it... at the time I was upset, now I'm glad I escaped this event. It's sad to hear even Japan is becoming a dangerous place for internet freedom and has a government failing to understand technology and the flow of information. I know mastodon.cloud was one of the largest instances out there. If it can go down, the fediverse as a whole is at serious threat of significant data loss. That's why I would further implore that we please support FULL account migration now... even if as an option disabled by default which the instance admin can enable if they so wish, in case performance is such a big concern. An equally better idea would be a node-based approach to storage, similarly to how IPFS / DAT in concept; Instead of an instance being stored on and served by one physical server, it can be stored in a decentralized cloud where anyone may run a node to serve a copy of the data. Sadly this would require such a rewrite that we'd be talking about a whole new project over the existing Mastodon. |
|
A reminder that we're supposed to "Own Your Own Data With Mastodon!" If we don't find a way to do that with toots, we are bound to be sued for false advertising. I smell a frivolous lawsuit that's bound to cripple the Fediverse and we need to get ahead of it. While I may not ideologically agree with Gab, which accounts for 25% of the Fediverse's mass, they have every right to be here as we do, and we need to prepare for pointed media smear campaigns that could potentially harm our reputation. We need to emphasize our content controls, and remind users that they can ban who they want from their own console, and that moderators do not have to do it for them. We have that advantage over Twitter and we need to advertise th out of it. |
Mastodon's own code is under the AGPL, right? That's the situation you're talking about, right? I'll agree the AGPL would have no effect on media outlets saying whatever charged things about mastodon they want to, though. |
|
That is an interesting point. I think they forked Mastodon, though (as of 2.8.5) and created Gab out of it. Ironically, it seems they were able to preserve their "toots" during the migration. My comment revolved around the idea that you can "Own Your Own Data With Mastodon". If this is the case, we must ensure that people have the capability to export their toots otherwise that slogan could be misconstrued as false advertising. We must have a provision that allows users to migrate to another server if the one they're on is being shut down through no fault of its own (regulatory crackdowns in the case of Mastodon.cloud). What happened in Minneapolis is a distraction. The attacks against the Fediverse and all other entities like it will intensify. We must fortify our infrastructure and address any outstanding issues so that we're ready for any exodus. |
|
Even a static representation of said toots is fine with me, similar to Twitter's archive, which you can navigate offline. They don't need to be integrated into the post chronology. Perhaps we could create a designation for such toots, and also prohibit their editing or removal to ensure a smooth migration. |
|
there's an issue for that #9461 (export including some bare html-version of your toots) With regard to european GDPR, there's the rule that you must have access to a machine readable version of you data. This is satisfied already. While this satisfies the letter of this particular law, I do wish it was easier for everyone to use that data (hence the ticket above). I also think re-importing would be nice. Yet, I do not believe there's any grounds to your "false advertising" point. Sorry. (edit: I linked to the wrong issue before, this has been corrected.) |
|
Thank you for clarifying that. I'll follow that issue too. ^_^ |
|
I strongly support this. I have my own independent instance that I don't want to maintain anymore so I moved over to another instance. DigitalOcean doesn't let you download complete images as a backup so I can't wait to upgrade to a later version that supports it and transfer the old content over. |
|
This is a feature that it's impressive that Mastodon does not have in 2020, honestly. Without the ability to export and import post history not only can accounts be lost but also identities - as even if an account is migrated to another server, at the moment this is a partial "in name only" process. Any content that the old account had posted is still bound to the lifetime of the old service (and its FQDN). After having tested some scripts for extracting the information from the exported data dumps, I think there's enough of those scripts and projects wandering about that it's perfectly feasible to grab one (1), integrate it into Mastodon, and offer it as an utility that eg.: generates full sessions / scripts for clients like |
|
@lmachucab do the existing scripts deal with not treating the imported posts as new posts that would be picked up for federation? From what I'm reading above, that's what should be avoided to reduce load on servers. I would be happy if my posts were only viewable from my account page/feed rather than immediately (or ever) federated (i.e., they would only be federated if someone was looking at my account). All I want to be able to do is change my username or move to another instance if needed. |
|
I thought I'd bring this up: One of the SPC admins revived bofa.lol, a previously discontinued instance, only for it to regurgitate its old content over TWKN, some of which was excessively vile. bofa.lol aside, I'd like to focus on the mechanism that caused those toots to refederate and see if we can harness that mechanism to archive them for other instances. As of this writing, bofa.lol was taken down less than a day after it was briefly revived. *Edited for context and some spelling mistakes at 21:50-0500 |
2021* |
|
This is going to be incredibly important to ensuring Fediverse platforms don't centralize around the most popular instances. If I have the ability to completely move my account from one instance to another with minor hassle, chances are I'm going to be more willing to take risks on smaller instances. Some of my first accounts on Masto/Peertube were on small instances, but they ended up closing down on me unexpectedly. |
|
@Gargron, do you consider this to be a feature worth pursuing? if so, what is needed to make it happen? it's been open for nearly 2 years without any comments from mastodon's lead, so i thought i would ask. |
frankly I think this is the most reasonable behavior considering the exponential complexity of doing more. people are migrating from twitter as we speak and failing to act for another 7 years risks a massive amount of data disappearing. perfect is the enemy of good enough |
|
It seems to me under this situation, moving server would preserve your history, but not your relationships, identity or reputation. For the system, you're essentially a new user with new old messages ? I understand that is better than nothing. But it also means leaving your server, you will become nobody again, as far as any the rest of the system is concerned. And you would have to rebuild your social network and your place on other people's feed, from the ground up. In other words, if you actually care about building your online presence, that would still be the end of it. Also, all these posts getting new message IDs, they're not all going to get pushed out to the federated network ? Just creating internally but unknown to any outside server until they ask about your history ? (re-publishing, potentially years worth of user history and interactions would flood the network, even if it's back dated) I think that all could work, but there need to be a seamless link between old account to new account relationship. Like posting a user account obituary to the whole federated network olduser@olderserver.com is now newuser@newserver.com, update your local databases to re-write all olduser@olderserver.com mentions to newuser@newserver.com, including private messages on all servers that have ever interacted on olduser@olderserver.com. Also doing this should be signed by the private key certificate of olduser@olderserver.com, do mastodon useraccounts have a private key certificate to sign such a document ? |
yes. this is a well-known problem with the Web. it's called "link rot". redirects require the old URI to still respond to requests. this works for when a browser or user-agent requests the old link, but it doesn't magically rewrite every single old resource or document. there's no way to assert to every known server that you're redirecting or have moved any given resource. especially not after the original server goes down or stops responding to requests. even with static html or whatever, "moving" the document to a new identifier, it becomes a completely different document , it just happens to contain the exact same contents. there is no external identifier for these resources. there is only the URI, which depends on the domain and the path not changing. it's a Very Hard Problem to solve for the Web. so, once again, the most that can be done is as described above: #12423 (comment) |
|
But there is a solution, have every server on the fediverse, looking into their databases and overwrite every mention of olduser@olderserver.com to newuser@newserver.com "hey, every server, olduser@olderserver.com is now newuser@newserver.com, update all records and URI, I sign this message with private key of olduser@olderserver.com 1986htg8h1308yn13g0hg" Now, I think it would have been better if the server saved a lookup table for each user, and they only changed the destination olduser@olderserver.com in the lookup table, instead of every single piece of content associated with olduser@olderserver.com. But I guess the system wasn't built that way |
that is a LOT of compute to duplicate across the entire network. nevermind the potential for horrifying glitches if say, the operation gets interrupted for whatever reason I also want to stress this problem does not need to be solved at the network/protocol level. frankly, just letting the wayback machine do what it has always done would provide a reasonable way for researchers (and let’s be real, that’s the only group that regularly looks at other people’s > 1 year old posts) to resolve broken links |
adding onto this, when we ask for features I think we need to be honest with ourselves about one important thing: we are asking other people to do work for free. important work, work that has the potential to positively effect society in a real way, but we need to be empathetic about what we ask for. unless you can find a core contributor who is really gung ho about solving this complex problem the world wide web and email both accepted as a necessary evil, I don’t think demanding more than the minimum viable solution is something that should be taken seriously at this point. people have lives to live and that needs to have weight in these discussions |
It is possible to have an imperfect solution that uses the user-level message ID to find the corresponding instance message ID. Mastodon already has a user redirection feature, so similar functionality is theoretically possible until the old server stops working. A user moves from A to server B. Server C tries to get a particular message from A and is told that the user went to B, and that the corresponding user-level message ID is XXXX. C then goes to B and uses the user's XXXX message ID to find the corresponding instance message ID, and then makes the corresponding action. Neither user redirection nor message redirection are specified by the ActivityPub protocol. So you will still lose redirects to other incompatible servers.
That sadly, link rot happened everyday.
You can't keep them as that will result in 2 or more users trying to occupy the same ID. |
|
Hey all, I'm new here but I recognize the importance of this project and would love to jump in and help out however I can. Looks to me like we've got a pretty solid consensus around @RinCat 's suggested approach posted a few days ago. Sounds like some specifics still need to be worked out... but do you think we should go ahead and create a feature branch for this to keep the momentum going? |
|
On a side note, I think the time has come to make a serious push on this. Earlier today I was reading about Bluesky and the AT Protocol (sorry if these are dirty words here) and saw that they are strongly emphasizing "account portability" as a key advantage of AT over ActivityPub. So I think addressing this issue should be a very high priority. The idea of a separate and (intentionally) mutually incompatible Fediverse funded by corporate money is a scary possibility; in fact I see Bluesky and the AT Protocol as a major threat to Mastodon adoption in the next few years. So I'd like to see these issues fixed and I'm willing to contribute if it can help make that happen. |
|
doing more reading this morning into atproto's "solution" for this and it appears that they are simply using storing a user's data in "signed data repositories", one of which is kept on the host server and one that is backed up to the client periodically. In their own words:
This is hardly groundbreaking stuff; I'd imagine that we could come up with something similar on our end, or certain apps or clients could find a way to implement this automatically for users without them needing to do a manual backup. Maybe volunteers could also run datastores separate from the primary mastodon servers to provide automated cloud backups of these records (similar to how Mastodon servers are community run). These could be independent of servers driving the mastodon core functionality and also independent of the hosts of those servers, so that a mastodon server going down or one bad admin doesnt mean all your content is lost forever. Regarding privacy, maybe we provide the option to encrypt backup records such that a user is the only one who can recover their own data. And maybe we provide additional options for data backup for superusers who do not want to use this method for whatever reason? Open to comments or suggestions on any of these ofc. |
|
Regarding restoring one's followers, could it help to have a system where when you restore your data, followers are automatically restored where possible following a verification or authentication procedure of some sort? |
|
@jimbopolous the entire lynchpin allowing any portability in bluesky/atproto is that they use a centralized name server: the Placeholder Server mints and resolves |
|
@trwnh wait so if that's the case it would mean atproto isnt truly decentralized right? Even so, I think your average social media user will not understand the difference and it could be the deciding factor for less-informed users who choose to adopt Bluesky instead. atproto also specifically calls out ActivityPub claiming that it does not have good enough account portability... and it seems to be the main concern that is preventing many people I know from committing to Mastodon. So IMHO addressing these claims directly should still be a high priority. ( @trwnh I think I may need to do some more research on how DID works to understand your point better, so I can do that over the next few days). |
|
@jimbopolous nope, it's a centralized network with decentralized data storage. so basically like github. you can use did:web, which is basically like https:, but it comes with the same characteristics as any other web-based identifier, and it has the same issues: https://atproto.com/specs/did
|
|
I think the proper place of user data is ultimately on the user's device. This data can be cached on servers for redistribution (in the same way that twitch, caches and rebroadcast your stream) And moderation would not be done by deleting user content but by broadcasting a filter against them, like a black list and by ceasing to rebroadcast their content. In that optic, I think the idea of a single user mastodon instance is the most realistic way to get to that. The process of creating such an instance on your phone or computer should be streamlined to the efficiency level of installing an app and choosing a name. Federation should happen automatically and all default should be sane for them being good fedizens. The current status quo of instance owners offering free cloud storage and mastodon infrastructure for user's toots, should be seen as an undesirable growing pain of the fediverse/mastodon. It exists because of current excess complexity of creating a mastodon instance. Not because the user lacks the storage, processing power, bandwidth or even the connectivity since your most basic of cell phone has more than enough of all of these to server as an adequate single user instance. The only thing a user might want out of cloud infrastructure is caching, so that their phone's data plan doesn't get eaten by sending their cat's picture to a million stranger. This data should just be store and forwarded accross the fediverse and an offline user's interaction be stored and queued until they come back in a manner similar to the MQTT protocol. As far as I'm concerned, I would rather only use clouds only for rain and use my own computers for everything else. |
|
@shodanx2 Distributed networks have a lot of challenges! And they're not easy! For instance, mastodon servers are nearly always online, so they can just queue shit and throw them at each other and then eventually give up, and it's enough A cell phone could never run an instance, because it has to much downtime! For a fully distributed approach, you could check put Scuttlebutt. What would be nice if Activitypub Client-To-Server was more prevalent, which would encourage and even require Apps to keep track of the entire Profile and all recieved Messages. Like a BlobDelivery Activity containing a signal protocol message? Which would expand into a Create or Add activity? But as it stands, the server has all the responsibility and all the data, and at least for mastodon, i don't see that changing. You are asking for fundamental and radical changes! |
|
@shodanx2 I think it would be awesome if mastodon could provide what you're describing as an option for users. I also think it's important to accommodate users who are unable or unwilling to self-host for whatever reason. Based on what I know now I'd like to someday see an approach like this: "Your data belongs to you. Here are a few options we've made easy for you. Choose whichever one best meets your needs." However, right now I think the problems we should focus on are:
Hopefully switching servers will be a relatively rare event for most people so I think it would be ok if early implementations require users to do a little bit of documentation-reading. In the meantime if a crisis occurs, like a major server going offline with a lot of stranded users, we can find ways to mobilize some volunteers to provide assistance with restoring accounts. Trying not to have any strong opinions until doing some more research first, but I'll keep monitoring this thread's replies and report back once I'm a bit more prepared to weigh in on implementation specifics. |
|
I'm giving up on this issue being implemented on Mastodon, and looking into Calckey or Takahe or others instead |
|
@DRKSLV these are good points as well. The great thing about the Fediverse is that people with strong privacy needs can just use a different app specializing in that and still participate. One of the goals of Mastodon seems to be mass-adoption so it still needs to be intuitive enough to be used by the average smartphone-owner. so I think whatever solution we implement should stay true to that. |
|
Anyone else here aware of the Solid project? It's still pretty new, and making Mastodon compatible with it would likely be no small feat... but I've been doing some research and I'm beginning to wonder if it could elegantly solve a lot of the problems being discussed here. Link to the Solid Project site here. Any thoughts? |
|
A discussion about Solid may be a bit off-topic for this thread, so I've created a separate discussion post to explore further if anyone's interested: |
|
Sorry for monopolizing the discussion but I've thrown together a quick diagram to demonstrate how I think Solid could solve Mastodon's account migration challenges. General reactions welcome in this issue thread, but for more in-depth discussion we may want to jump over to the conversation thread I've created so this chat doesn't go off the rails: #26403
|
|
I think at a high level this approach makes sense but I'm still digging into the low-level implementation specifics. If anyone is aware of reasons why this would not work please feel free to share those as well. In the meantime it looks like there is at least some support for this design so I will start working on a POC on my end. If anyone wants to work with me on this feel free to DM me or reach out via the discussion thread: ##26403. Will report back here when I have some results worth discussing. Also I have created a fork and named it mastodon-solid-poc, if anyone wants to check it out. Thanks! |
|
Hello, any update on this feature? It could be great to be able to move everything in case of an issue with a server... |
|
hi @juliennnnn , I've been looking into the approach that I suggested in my above messages. I'm making progress on the research side but there is a lot to learn and I have not started implementing anything yet. My goal is to have a POC complete by the end of 2023 and by then we should better understand whether this approach is feasible. I'm also looking for others who are interested in working on this, so if you know anyone it please feel free to put them in touch with me. It might help speed things up. |
|
Apologies for the delay in sharing this here. A development/test version of the solution outlined here is now available as an external command-line tool, at https://mastodoncontentmover.github.io/ It's not as sophisticated a solution as many in this thread are looking for, but it provides basic functionality to move post content, including media attachments, for people who need to migrate between instances. It does not preserve threaded interactions, but it does preserve self-reply threads (involving threads where all posts are by the account owner themselves). It respects the Mastodon standard API rate limits with an additional margin, and it is possible for users to optionally slow the tool's activity even more. Media posts in particular are significantly throttled because media processing is resource intensive, and by default it suppresses public posts so that timelines are not flooded. It's possible to selectively save and/or repost only bookmarked posts. It runs on any platform that has a Java runtime environment (version 1.7 or above), and the source code is available on Github. It is only intended as a workaround; I built it because I needed to move my own content, but was mindful throughout that this functionality was also needed by others — it makes sense to share it, but because of how it came to be and because of time constraints it is a little rough around the edges. Nonetheless, I hope it helps some of those facing the loss of their content due to the absence of this functionality from Mastodon itself, and ultimately I hope it might prompt Mastodon to consider building at least this basic level of functionality into core (which would offer some advantages such as allowing admins more control over post importing and how that is prioritised on their instance, and possibly allowing posts to be correctly backdated as well). |
|
Really fantastic to see some progress toward a content migration solution! Thank you for sharing your work. |
|
I favor a very simple ‘forwarding address’ approach that leans on existing federation features as a form of archiving:
|
|
@MastodonContentMover this looks great! Will definitely help a lot of people solve this problem for themselves without requiring any major changes to mastodon's architecture. I'm also continuing to work on the solution I described in my earlier posts. I'm hoping it will provide a seamless solution to the Fediverse's issues with account and post migration in general. I've gotten started on it and have figured out a lot of the implementation details, but have been struggling to implement them due to my poor understanding of OAuth 2.0 and OIDC. I'm brushing up on that at the moment. For more regular updates or if you want to get involved, follow the discussion thread I've started here. |
and others
#177 – Support Account Migration – was closed after implementing follower migration, but this is only one small part of a true migration. To really be able to change instances, you need to be able to take your posts with you. There was some good discussion on this over there; I'm opening a new issue to make it clear that this is a separate concern from that issue, which seemed to evolve into only being about followers.
Personally, I couldn't care less about migrating my followers/following list. I can refollow people and have them refollow me. It'll all work out. My posts, however, are currently impossible to restore.
The text was updated successfully, but these errors were encountered: