Builds source tarballs dynamically

[conorsch]

Ready for review. For reference, see discussion in #182 (comment)

Changes

Full backwards compatibility is retained, so the original invocation will work just fine:

PKG_VERSION=0.2.1 PKG_PATH=tarballs/securedrop-client-0.2.1.tar.gz make securedrop-client

There's now an optional, terser step to build a package, where the version defaults to the latest tag, and the tarball is generated dynamically from that tag:

make securedrop-client

Regardless of which method is used, the .deb file generated from it is byte-for-byte identical over multiple runs. In both cases, the timestamp information is pulled in from the package changelog. Ideally we'd be using SOURCE_DATE_EPOCH in all places to manage the timestamp info, but not all tools support it yet (particularly not 'python setup.py sdist'). Hopefully we'll see movement upstream soon, and then we can standardize on using SOURCE_DATE_EPOCH everywhere to control timestamps.

It's possible we'll want to break out the logic that chains python-sdist -> git -> tar -> gz so we can reuse it elsewhere, but not doing that now.

Testing

debs are reproducible

Regardless of how the tarball was generated, or even which was used, the resulting .deb file for a specific package version should be identical. Confirm that by running the following:

#!/bin/bash
set -e

results_dir=/tmp/test_results
rm -rf "$results_dir"
mkdir -p "$results_dir"

echo "Building from new dynamic logic"
make securedrop-client
cp /home/user/debbuild/packaging/securedrop-client_0.2.1+buster_all.deb "${results_dir}/dynamic.deb"

echo "Building from historical static tarball"
PKG_VERSION=0.2.1 PKG_PATH=tarballs/securedrop-client-0.2.1.tar.gz make securedrop-client
cp /home/user/debbuild/packaging/securedrop-client_0.2.1+buster_all.deb "${results_dir}/static.deb"

echo "Comparing results"
sha256sum ${results_dir}/*
diffoscope ${results_dir}/* && echo "SUCCESS: fully reproducible"

You should see output of 4b65f8a91f8555a2e0303ce2cdb97a58addddbc7a2e6d3567805b8aa0a0f4ed3, with the "SUCCESS" message printed at the bottom.

tarballs are reproducible

The tarballs are an intermediate artifact in our build process. Let's confirm that if we build them dynamically from source, we're getting identical artifacts—which is a new feature introduced by this PR.

# Prepare a results dir
rm -rf /tmp/results
mkdir /tmp/results

# Use the new dynamic build logic
make securedrop-client
cp /tmp/securedrop-client/dist/*.tar.gz /tmp/results/1.tar.gz
rm -rf /tmp/securedrop-client

# Run it again
make securedrop-client
cp /tmp/securedrop-client/dist/*.tar.gz /tmp/results/2.tar.gz
rm -rf /tmp/securedrop-client

# Compare the results!
sha256sum /tmp/results/*.tar.gz
diffoscope /tmp/results/*.tar.gz && echo "SUCCESS: fully reproducible

[sssoleileraaa]

this is a good idea. will your change still support supplied version numbers in case we're building a dev release?

[conorsch]

Yes, fully backwards compatible in that PKG_VERSION & PKG_PATH can still be provided as overrides, they're just not required. The verification logic needs some work before it's ready for full review—at least:

1. use a dedicated keyring so that only the prod signing key is used to validate tags
2. detect if an "rc" tag is being used, and if so, don't expect a prod sig

Then it should be ready to bang around on.

[kushaldas]

Can we please add some error messages here somehow? At least for missing key or expired key :)
I was wondering what is wrong for some time with the following output.

✦ ❯ make securedrop-client
		PKG_NAME="securedrop-client" ./scripts/build-debianpackage
	gpg: key 310F561200F4AD77: public key "SecureDrop Release Signing Key" imported
	gpg: Total number processed: 1
	gpg:               imported: 1
	gpg: Signature made Tuesday 28 April 2020 07:57:53 PM IST
	gpg:                using RSA key 22245C81E3BAEB4138B36061310F561200F4AD77
	gpg: Good signature from "SecureDrop Release Signing Key" [unknown]
	gpg:                 aka "SecureDrop Release Signing Key <securedrop-release-key@freedom.press>" [unknown]
	gpg: WARNING: This key is not certified with a trusted signature!
	gpg:          There is no indication that the signature belongs to the owner.
	Primary key fingerprint: 2224 5C81 E3BA EB41 38B3  6061 310F 5612 00F4 AD77
	Checking that SHA256SUMs from mirror match signed file...  OK
	PKG_VERSION not set, inferring from recent releases...
	Using PKG_VERSION: 0.2.1
	securedrop-client is a Python package
	PKG_PATH not set, building from source (version 0.2.1)...
	Cloning into '/tmp/securedrop-client'...
	remote: Enumerating objects: 64, done.
	remote: Counting objects: 100% (64/64), done.
	remote: Compressing objects: 100% (48/48), done.
	remote: Total 9276 (delta 32), reused 26 (delta 16), pack-reused 9212
	Receiving objects: 100% (9276/9276), 6.95 MiB | 2.51 MiB/s, done.
	Resolving deltas: 100% (6863/6863), done.
	make: *** [Makefile:9: securedrop-client] Error 1

[conorsch]

Great point, @kushaldas. Added. Try again and confirm that an error message displays for you.

[kushaldas]

• Deb packages are reproducible
• tarballs are reproducible

This is amazing, approved. I will still open a separate issue for a better error messages related to GPG.