Better distribution of deb packages via a repository

[jinnatar]

Problem statement

While the oneliner installer is nice and can be re-run for updates, it suffers in that one has to know and remember to go re-run it, and find through their history how they ran it the last time. The solution is packages installed through standard or unofficial package managers.

For Debian derivatives the releases do contain a prebuilt deb and in the past there's been interest in getting that distributed through apt repositories, but the process and policy of official Debian repos is Hard and not worth the limited effort available.

Proposed solution

A project specific package repository owned, operated and signed by project owners is a fairly common and well supported method of deb distribution and is directly compatible with apt. There's ready-made tooling to turn debs & a signing key into respository files that are then hosted on any static website.

Demo repository

To demonstrate the tooling needed to easily create and update such a repository and have it hosted on GitHub Pages, I've borrowed liberally from what's been done in https://github.com/kanidm/kanidm_ppa and stood up https://github.com/jinnatar/atuin_demo_ppa. The public facing view of it via GitHub Pages is: https://jinnatar.github.io/atuin_demo_ppa/

• I did not include the GitHub Actions workflow kanidm has because it requires entrusting your GPG private key used for signing to GitHub. The entire process can be automated via an Actions workflow, but trusting GH is the caveat.
• On update the scripts pull the latest GitHub release in atuinsh/atuin and any deb packages on it as artifacts.
• Once you have a GPG signing key created / imported locally, cutting a signed repo update is as easy as:

GPG_KEY_ID=yourGpgKeyIdHere ./update.sh
git add debian/
git commit -m "Push latest release"
git push

How could this be official?

• The easiest way to replicate would be to create a similar repo under the atuinsh org and link it's main branch to GH Pages.
• Ye need signing keys, and to store the public key in KEY.gpg. It can be generated with gpg --armor --export $GPG_KEY_ID > KEY.gpg. There's so many best practices around that that I won't pretend to be able to give advice on it. The script supports using a subkey as best as I could test it, so even the most paranoid secure options should work here.
• My recommendation would be to start simple, i.e. not bother with GH Actions and update the repo maually after cutting the GH Release in the main repo. On a machine that has the signing keys it's just one command and doing the commit & push.
• Right now only amd64 packages are attached to releases. If others are added, the repo scripts can already handle that.

Obviously if we find an official path, I'll decom the demo repo. Already using it myself on several Debian 12 machines and it's working beautifully from a user perspective. The problem with it is that it's signed with my keys, so the packages are only as trustworthy as anyone is willing to trust myself. :-)

[jinnatar]

One important caveat though that I failed to mention! The script uses Debian specific tools such as dpkg-scanpackages & apt-ftparchive that are probably annoying to get on a non-Debian based system. So one probably needs a layer of a Debian container to run the releases from any machine. I'll see about adding a quick Dockerfile to have an easy environment for that.

[arcuru]

It looks like there are three different projects following this to setup their own PPAs, and it would likely be useful to many more. Do you feel it would be easier if there was a coordinated PPA that contained multiple packages? Or should they all just use this as a template to setup their own?

Basically if it makes sense at all to setup a single 3rd party repo that was easier to add packages to than the official Debian ones.

[jinnatar]

Content warning: veering heavily off-topic and into opinion as well, but the question is worth covering.

There's a handful of problems with shared repos or repo collections I'm aware of, but am in no way an expert on the matter.

• For apt based systems, individual packages are not signed, and instead the repository delivering them is, which then provides the mechanism to prove the packages the user got are the same ones that were ingested into the repo. This does not automatically prove the package is the same one that the originating project published. Ergo, a user needs to trust the repository owner. This could be improved, but goes way into advanced territory from the point of view of project owners that want their stuff securely distributed.
• There is no standard method of ingesting packages into a repo that I know of or can find, well beyond the process Debian has for their official repos. They're built as a one-way channel. So any method of populating said repo would be specific to it.

There is one "official" way to do PPAs, which is Launchpad: https://launchpad.net/ubuntu/+ppas .. For example see https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable. They also have issues though:

• They do solve the problem of separate trust per project by hosting multiple per-user repositories with their own signing keys. (spoiler alert: they use really bad keys though)
• Very geared towards Ubuntu, but is available and in theory works on other Debian variants.
• Does not publish repository public keys as a file (that I can find), you are reliant on keyservers to have and give you the key. This is usually fine, but requires publishing the signing key to said keyserver, and doesn't always play well with other software.
• add-apt-repository sucks. Personal opinion, but corroborated by weeeeird issues no one should have: https://askubuntu.com/questions/1480616/adding-opencpn-repository-attributeerror-nonetype-object-has-no-attribute .. (I ran into this just now testing it for the first time in a while!)
• Launchpad has a terrible track record when it comes to security, as an example see: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1461834 .. they require insecure keys, holding everyone back from using more secure ones.
• If Canonical can't do PPA hosting correctly, there's probably not a long list of people willing to try doing better. :-) .. It's easy pointing fingers from the outside how bad they are, but probably it's a bit of a nightmare in any case.

While it would be great that we had something like crates.io, pypi.org flathub.org, homebrew, chocolatey etcetera, the fact is that Debian predates all of them and no one seems to have stepped up to make a more radical shift within that ecosystem. Of course we could all move to flatpaks and just ignore the whole problem. That's definitely how I deal with Discord being annoying to install and update. :-)

Long story short, I prefer and propose the KISS approach. A per project repo tightly controlled by the project owners provides the best guarantees after official repos and is still relatively low maintenance, given the right tooling. I was a bit surprised how simple it is in the end, the only hard part is doing key management (which you'd need to do anyway) and hosting the damn files somewhere stable.

[jinnatar]

FYI, I finally did finish the RELEASE_README with Docker instructions to make it easier to cut a release even if one does not use Debian: https://jinnatar.github.io/atuin_demo_ppa/RELEASE_README