[security] switch away from self-hosted packages

[cyphar]

Currently, in order for a user to run their own homeserver they are told to use the package repos hosted by matrix.org. This is not a great idea, for a few reasons (which were highlighted by this attack):

1. Single point of failure. During the attack and rebuild, the package repos were down -- so users wouldn't be able to install new packages. An example of this is that I have a Matrix homeserver set up, but since riot.im was down I wanted to install riot-web in order to have my own riot-web client. Unfortunately the packages are hosted on riot.im, so I'm SoL until the infrastructure is back.

2. There is no separation (from a security perspective) between the package hosting and homeserver hosting. Yes, you might have firewalled things -- but since the same people are in charge of the infrastructure of both then an attack on the Matrix homeserver can cause issues with the packages used by self-hosters and vice-versa.

Ideally, I would suggest that packages should be entirely maintained in distributions. But at the very least they should not be hosted on the matrix.org infrastructure. I'm a bit biased, but I would suggest hosting on the Open Build Service which supports hosting for a vast array of distributions and is entirely maintained by SUSE/openSUSE. And GPG key leaks like this wouldn't happen (users don't have access to the keys). Yes, you'd have more credentials to deal with but the building of packages is entirely transparent on OBS (rather than being a random blob we download from matrix.org).

[ptman]

At least debian https://packages.debian.org/sid/matrix-synapse and ubuntu https://packages.ubuntu.com/disco/matrix-synapse have synapse packages. They aren't the latest, but at least debian has a security team that looks after stuff.

[cyphar]

As does openSUSE. I think distributions packaging it themselves is the most ideal solution for users (since it gives extra chances for review), but users should either be told to use the official packages from distros or the packages should be hosted outside of the infrastructure that hosts matrix.org.

[gloomy-ghost]

separation (from a security perspective)

as long as the signing key is air-gapped it doesn't really matter where the package is hosted. the problem is - do you trust matrix.org folks are following this practice.

[cyphar]

as long as the signing key is air-gapped it doesn't really matter where the package is hosted.

Sure, though as I mentioned above you still a single point-of-failure on the Matrix.org infrastructure (so maybe you won't get a malicious package, you'll just get a 404). There's also the point that maintaining the package in a distribution gives you extra review steps (rather than being something that's automatically pushed from CI).

the problem is - do you trust matrix.org folks are following this practice.

Well, we now know they weren't. To be fair, I'm sure the vast majority of third-party repositories have undesirable setups for their package signing -- though it doesn't make the situation any less awful.

[Mikaela]

How about priorising Flatpak and Snap? Flatpak is only for GUI apps so it would only help Riot, but Snap also works for daemons/headless systems and I think IPFS is also distributed through it.

[ptman]

Docker already serves that purpose for synapse IMO.

[gloomy-ghost]

snap doesn't even let you disable auto-update... suppose an update deadline can force people to audit the package before installing it?

[afranke]

This is filed against the repository for the matrix.org website. I’ll leave aside packaging considerations, which should be discussed elsewhere, and I’ll focus on what we can actually fix here: documentation and pages on the matrix.org website.

Currently, in order for a user to run their own homeserver they are told to use the package repos hosted by matrix.org. This is not a great idea, for a few reasons (which were highlighted by this attack)

For this to be actionable, it would have made our task way easier if instead of a generic statement like this we had an actual link to and quote of the place where “users are told tu use the repos”. I went to the homepage, then followed Discover → Synapse installation, to land on the Synapse installation guide. This in turn sends me to this documentation where I’m told to either get the package from PyPI (which makes sense as a Python package), or Docker images from docker.com, or to get it the one from my distribution, with many of them listed.

This is not surprising given this issue is over two years old. It seems obsolete now, so I’ll close it.

[cyphar]

This website which is now linked from the synapse README didn't exist in 2019, from memory the recommendations in the synapse repo for Debian users was to use the Debian repository hosted by the Matrix org.

However, that site (while it does explain how to build and run it from source) does still provide links to the matrix.org rep. In fact, in the "downstream repository" sections for Debian and Ubuntu the recommendation is to use the matrix.org repos:

We do not recommend using the packages from the default Debian buster repository at this time, as they are old and suffer from known security vulnerabilities. You can install the latest version of Synapse from our repository or from buster-backports. Please see the Debian documentation for information on how to use backports.

We do not recommend using the packages in the default Ubuntu repository at this time, as they are old and suffer from known security vulnerabilities. The latest version of Synapse can be installed from our repository.

But at least now the documentation does describe how to use distribution-maintained repositories (with the exception of Debian and Ubuntu which are still recommended against). I don't think there's much else to fix since the Debian and Ubuntu outdated package issue is not something that Matrix can solve, I just wanted to point out that the recommendation to use the Matrix.org repos for those two (very popular) distributions is still there.

As for why I opened this against the matrix.org site, it wasn't clear where else to open an issue with repos that are hosted on matrix.org -- it's not really a bug in synapse that matrix.org had a security breach.