Organizational issues arising from the xz supply chain attack

[EricSoroos]

Recently there was a supply chain attack on the xz package -- CVE 2024-3094 (https://www.openwall.com/lists/oss-security/2024/03/29/4) from a trusted maintainer, intended to backdoor ssh in Debian and Redhat based distros. It was a long process, involving a maintainer making changes to the xz distribution over more than a year.

In short, a new(ish) trusted maintainer made tarball releases that included attack code that was not in the git repo, that used binary test artifacts to affect the build process. The original maintainer brought on this new maintainer because he was overwhelmed.

There are a few factors in the ability to release and obscure this attack:

1. a single active maintainer
2. release artifacts that were not traceable to the git repository
3. opaque binary test data on the build system
4. a complicated build system

Looking at these items, we're in pretty good shape for three of them.

• We have a core team of 8 members. Pull requests and activity on the main and versioned branches are watched.
• our test data is python scripts, though there are some vendored javascripts and images.
• Python artifacts are nowhere near as complicated to build as system libraries.

The release artifacts are one place where we can definitely improve the transparency -- as they're currently built on a developer system.

• Python Package
• Docker Images
• Debian .deb package

Ideally, there should be a reproducible build and upload path using CI (likely Github Actions) where the code that is executed to build and upload is entirely specified by the contents of the repository. This protects against both an untrustworthy maintainer and a compromised developer machine. It has a nice side benefit of making the actual release easier to perform.

PyPi and Docker both have relatively straightforward methods to run GH Actions and upload. I'm less sure about uploads to packaging.ckan.org.

[amercader]

Adding some details on tasks needed for each of the areas we want to automate as raised by @EricSoroos :

• PyPi: This seems straightforward: Using the PyPI publish action plus setting up Trusted Publishers on PyPI, perhaps in a separate GitHub environment to manually approve the workflow run. The action would be triggered when pushing a new release tag.
• Docker Images: We are already building and pushing the master image nightly, we just need to copy the workflow but trigger it when a new tag is pushed to the ckan/ckan-docker-base repo (we need to udpate the Dockerfiles and README anyway, so we can push a new tag after a release)
• Deb packages: these require a bit more thought but essentially, building the deb packages is already automated via Ansible. These scripts are run locally inside Vagrant VMs. Vagrant inside GitHub Actions runners seems an abstraction too much (and besides it doesn't seem to be well supported), so I think our first approach should be to try to run the existing Ansible scripts in the relevant containers for the Ubuntu versions we support (going forward 22.04 and 24.04). We probably want to separate the building of the packages from the uploading to packaging.ckan.org (an S3 bucket), because we want to test the packages beforehand. Perhaps the action can generate the deb files as artifacts that can be downloaded and tested, and then there's a separate environment for building and uploading to S3. We can also consider hosting them in GitHub itself if it makes things easier.

cc @kowh-ai