Support Post Migration

[SilverWolf32]

#177 – Support Account Migration – was closed after implementing follower migration, but this is only one small part of a true migration. To really be able to change instances, you need to be able to take your posts with you. There was some good discussion on this over there; I'm opening a new issue to make it clear that this is a separate concern from that issue, which seemed to evolve into only being about followers.

Personally, I couldn't care less about migrating my followers/following list. I can refollow people and have them refollow me. It'll all work out. My posts, however, are currently impossible to restore.

[MirceaKitsune]

Once more I fully support this and have been eagerly awaiting the much requested feature. I would love to see the ability to import the data archives (favorites boosts and toots) exported from the https://instance.social/settings/export section, which is currently a one-way system only. Unfortunately there doesn't seem to be a plan for this yet, but I'm staying hopeful that will change eventually. I believe Eugen mentioned performance concerns on the previous discussion.

[trwnh]

Reiterating points from previous discussion:

• statuses are currently attached to an account, and need to be decoupled first
  • acct is currently authoritative over database id, so all internal logic needs to be reworked to use database id instead ([v5.0] Use local id consistently -- not just API routes, but for URIs and internal references as well #10745)
• status move should ideally be as simple as updating some records (after above points are addressed), but this can still be a heavy operation for the database
• status import cannot be backfilled for performance reasons

[SilverWolf32]

Thanks @trwnh!

status import cannot be backfilled for performance reasons

What's backfilling?

[trwnh]

backfilling is #34 -- fetching old statuses from a profile that were never delivered to the instance. you can't expect to deliver 100,000 activities every time you migrate, because that's wildly inefficient.

[MirceaKitsune]

Regarding performance and backfilling: This is why I believe that post migration is possible, but those who choose to use it will need to accept a long waiting period. There will probably have to be three types of limitations to make it viable:

1. An account queue. For example, the instance may only allow up to 5 accounts to be migrated simultaneously. The archive import menu should show a message saying something like "you have to wait until 10 more accounts finish migrating before we start the import process".
2. Every post and action will be migrated with a given delay. This could be one post per 10 seconds. The import menu should then show the message "10 out of 1000 posts migrated" to let you know about the status of the procedure.
3. Only a certain number of the latest activities will be imported. If an archive contains 1.000.000 actions, the interface informs the user that only the latest 100.000 will be considered.

Obviously, if you're importing something like 10.000 boosts + faves + toots, the process may take days if not weeks. It's a price the user will have to pay. But in this format it sounds like it should be performance viable at least.

[SilverWolf32]

That makes sense! Personally I'm perfectly fine with them taking a long while to move over, so long as they move over eventually.

I'm not quite sure what the limit should be, though – one post per 10 seconds seems awfully slow to me, unless you have like 100 accounts being migrated at once. Maybe set it relatively fast by default (like 1/s or maybe even a little higher), since most instances are small and won't need to deal with a lot of migrations, then larger instances can set it slower?

[SilverWolf32]

Although I would really, really prefer if everything gets migrated, even if it takes weeks or even months. I don't want to lose my old posts – that's basically an archive of what I'm like, so new people can get to know me a little better before they decide whether to follow me. It's also history, and while some may not want to keep that sort of thing, others (like me) would.

(Even if you technically have everything in the backup, that's incredibly cumbersome to look at, given I'm not aware of any software that can actually import it.)

[trwnh]

stuff can get imported, but it is absolutely a bad idea to redeliver it. i am firmly of the opinion that a Move with post import should maybe do a rewrite of existing statuses in the database, but nothing more. it is unwise to even consider backfilling until the actual decoupling and importing gets implemented.

more to the point in general, though: it is fundamentally bad idea to assume that every instance should have a copy of every single post at all times. we don't need to duplicate everything 6000x or more. what is fundamentally happening with a "migration" is that the authority is being shifted; the location is being shifted, that's all. it's a Bad Idea to treat them as new statuses. hopefully i've explained why

[SilverWolf32]

Hm...well, would they be new statuses? I figured they would simply be reassigned, but then maybe redelivered as the originals, not duplicated and "reposted".

Or maybe I'm completely misunderstanding how federation works. I know very little about the backend.

every instance should have a copy of every single post at all times. we don't need to duplicate everything 6000x or more

Is this actually being discussed? I'd say for a migration the situation is a bit different; only the destination server has to receive a copy of every single post, not everybody.

[trwnh]

only the destination server has to receive a copy of every single post

yeah, and it should be limited to this.

looking at prior art from zot, you can do an online migration (fetch old messages from your outbox and inbox to the new server) or an offline migration (using an exported account archive that contains at minimum your keys, your posts, and your address book)

[SilverWolf32]

That sounds lovely! I've been a little worried in the back of my mind about how to handle a server that just suddenly dies.

[lmachucab]

Is there at least a first-party tool that users (not servers) can use to work their downloaded backup and generate something usable? Something like a HTML file (plus media?) that can be uploaded into a remote server. Would at least temporarily solve the problem of making the lost content accessible again.

[MirceaKitsune]

@lmachucab I made a NodeJS script that I used to port my faves and boosts to other accounts. Unfortunately it no longer worked last time I tried it, and it was a crude solution that put quite a bit of stress on the target server and even triggered an instance's flood protections once. If you think you might have some use for it still, it's still available on my Github:

https://github.com/MirceaKitsune/mastodon_migrate

[masstransithonchkrow]

Hi there! Mastodon.cloud literally has a month to live. We're being shut down because the domain is being forced to bow to insane regulations, and the admin isn't having any of it.
https://mastodon.cloud/@TheAdmin/104227496654670309

We had been the target of immense DDoS attacks for the past six months.
If you need help or simply wish to deliberate, please let me know.