Synapse migration support

[TheSunCat]

Hey, thanks for starting this fork! Exciting to see this project continue.
I'm opening this issue to hopefully track implementation of migration support from Synapse, as I'd love to try out tuwunel once I know I can move my users to it.

[erebion]

Hey,

just wanted to add that even if migrating is not complete, it might already be helpful. I certainly don't care about losing messages, I just want to migrate to something that uses less resources.

For my use-case what I'd need to migrate, roughly:

• accounts
• room memberships
• room addresses

Even though there's currently no path for a full migration, could I somehow already just migrate that?

[criadoperez]

Hi! I started to work on migrating Synapse to Tuwnunel.
The work so far is here.

However its important to mention that current code uses an API + federation approach. Meaning it first exports the data of Synapse using the API, then imports it in Tuwunel and lastly uses federation. This approach makes it fast and safe, however it will not migrate everything. This file explains the limitations of this approach.

A more complete (but more complex and risky) approach is a direct database migration, which I will start to look into. If anybody else is also looking into this, feel free to reach out.

[RGBCube]

Thanks for the excellent work! A question I had in mind was, would it be possible to use Syanpse's database schema inside Tuwunel without modification? Or would it be a bad idea as the Synapse schema isn't stable?

[jevolk]

@criadoperez This is exactly what's needed to jumpstart this endeavor. Thank you so much 🫶. I'm going to have a look at this as soon as I return to work next week. One of the things I'm going to be aiming for is how we can get e2ee migrated seamlessly. I think it might just be possible to perform the migration of devices and access tokens in-situ from the Synapse server to Tuwunel. It would be disruptive enough if users had to reset their identities and lose their message histories that it's worth investigating this further.

@RGBCube Unfortunately no, though it's always been an option to plug into Synapse's pgsql using a rust-pgsql library to exfiltrate the data. Using the pgsql in production is way too complicated and suffers from a multitude of issues where rocksdb and sql simply don't work well together under the same abstraction.

[criadoperez]

I'm glad to see others are also interested in this. :-)

As a full DB migration is a completely different approach I've created a separate new repository to test this approach.

The general approach (simplified explanation) I have in mind is:

1. Dump the Synapse DB, keys and media files to a backup folder
2. Create in an isolated network a new tuwunel server that uses the same domain name and keys (this is important).
3. Create a "migration script that will query data from the SQL DB, adapt it and write it in the RocksDB format.

That last point is the real hard and tricky part, so I plan to implement it by stages. So for example, first only attempt to migrate a minimum viable room with no E2EE, media, invites, etc.

Then I can test in the isolated environment if that worked, and if it does, gradually attempt to add more and more complex information. That way we can have a table of what gets migrated and what doesn't.

We will also need a configuration script that can read a Synapse homeserver.yaml file and creates a minimum and equivalent tuwunel.toml file (plus copy in RocksDB some data that is in Synapse stores in the homeserver.yaml file but tuwunel in the DB)

[criadoperez]

Small update on this. I have stopped developing the "new repository" approach as it turned up to be more time consuming that I can dedicate to it. The original repository that does a more limited migration is there in case it helps anybody.

[PoneyClairDeLune]

Hello! Is there any progress on Synapse migration support?

[criadoperez]

As far as I know, nobody is working on this now I'm afraid. I considered creating a complete.codes contract to promote it but haven't done it yet.

[PoneyClairDeLune]

@criadoperez Thanks for the reply! I'm really looking forward to migrate from Synapse to get rid of it, hopefully it will get supported at some point.

Somewhat unrelated, I'm wondering why I got 4 downvotes for querying progress.

[Subject38]

I subscribed to this issue because I'm pretty much stuck using synapse until this is solved since i have a 25gb synapse db. This is almost certainly an issue that could be solved with an actionable bounty of a few thousand dollars though, hell I'd be more than willing to try my hand at it if there was some monetary incentive. Someone else would probably be willing to as well

[cadaniel]

@criadoperez Thanks for the reply! I'm really looking forward to migrate from Synapse to get rid of it, hopefully it will get supported at some point.

Somewhat unrelated, I'm wondering why I got 4 downvotes for querying progress.

As one of those downvotes, I'll let you know in a more kind way than many people on this site will. (at least in my experience, people jump down the throat quick here)

GitHub no longer shows how many people are watching an issue anymore, but there's probably dozens to several hundreds.

It's taken as bad form to ask for updates like this, as it notifies everyone with your question, that adds no value to a discussion, and if the developer had something to post, they likely would have.

From you it's a one off. But there are 68 open issues right now, and if one person in everyone periodically asks what's the status, it's a lot to manage and reply to.

Most open source devs are volunteers doing this in spare time. People like me would rather see the dev devoting time to actually working than replying to "is there any update" messages 4 times a week.

[railsmechanic]

When we migrated from Rocket.Chat to Matrix back then, we used the application rocketchat2matrix which works quite well. It's based on an AppService and - as far as I can see - uses Synapse admin endpoints /_synapse/admin/... and default matrix endpoints e.g. /_matrix/client/v3/rooms/... to migrate users, rooms, and messages. Perhaps this could serve as a potential blueprint for a migration tool? Of course, only if the messages aren't encrypted.

Edit:

Using rocketchat2matrix as blueprint, I've built (with Claude) a migration CLI tool. The first test worked well so far. I was able to import users, rooms, and messages. Of course, only unencrypted content. I intentionally skipped the uploads and such.

Once I've gone through it completely, I'll publish the source code.