Rebuild all wheels reproducibly and use wheels from local filesystem

[kushaldas]

This patch stops accessing the webbased index for the wheels,
instead uses the local filesystem path.

Fixes #212 (work remaining)

Next step in this PR.

• Someone with signing capability should regenerate all the wheels for securedrop-client, securedrop-proxy and securedrop-log projects. ./scripts/build-sync-wheels --clobber -p ../securedrop-client is one example.
• Then we should regenerate the build-requirements.txt files for all the 3 Debian packages
• CI should be green
• Remove our python package index.(temporarily via infra work)
• CI still should be green.

How to test?

• make securedrop-client
• make securedrop-proxy
• make securedrop-log
• CI should be green.

All of the above commands should pass.

[conorsch]

regenerate all the wheels

@kushaldas I've appended new wheel files and checksums here, and marked the task as complete in your list. Back over to you!

[eloquence]

(Retitled for clarity, please don't hesitate to edit if I misunderstood but I wanted to make it clearer that this PR now also updates all the existing wheels.)

[kushaldas]

Next, we will have to merge these 3 PRs:

• Uses new reproducible wheels from our builds securedrop-proxy#81
• Uses new reproducible wheels from our builds securedrop-client#1203
• Uses new reproducible wheels from our builds securedrop-log#21

Only after those are merged, we will be able to see CI progressing for this PR.

[conorsch]

@kushaldas I haven't provided in-depth review of all the checksums in the component PRs you link to, but plan to do so tomorrow. While testing that, I might append yet another test-only commit onto this PR, tracking the feature branches in those repos, to confirm everything is green prior to approving.

Thanks for preparing! Looking forward to landing these changes.

[kushaldas]

So the package build tests were failing as they were building the last release. And the repository now has different wheels from the last release. In my temporary commit, I am doing night build instead of rebuilding the last release.

I am still not sure how to fix the reproducibility-checks test for this purpose. @conorsch please let me know what do you think.

[conorsch]

And the repository now has different wheels from the last release.

From the last release of which package? Are you recommending that the wheels in this PR be regenerated to resolve?

[kushaldas]

So the package build tests were failing as they were building the last release. And the repository now has different wheels from the last release. In my temporary commit, I am doing night build instead of rebuilding the last release.

I am still not sure how to fix the reproducibility-checks test for this purpose. @conorsch please let me know what do you think.

And the repository now has different wheels from the last release.

From the last release of which package? Are you recommending that the wheels in this PR be regenerated to resolve?

• securedrop-client
• securedrop-proxy
• securedrop-log

Generally the CI is trying to build the last release of these packages, which is failing because now we have new wheels. My temporary commit tries to build the nightly, that is why is passing for those builds. But, the reproducibility-checks test tries to build from the old tarballs, which contains the old wheel checkcums. This is why https://circleci.com/gh/freedomofpress/securedrop-debian-packaging/6586?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link is still failing.

[emkll]

I am still not sure how to fix the reproducibility-checks test for this purpose.

@kushaldas you correctly identify that the reproducibility check rely on the archived tarballs[1], which were have considered deprecating. It sounds like we could replace this step by manually cloning and building the tarball locally on the latest release tag on main branch of the target repos using some of the logic introduced in [2]. You can then, for the purposes of this PR, add a temporary commit to check out the feature branch with the updated hashes.
[1] https://github.com/freedomofpress/securedrop-debian-packaging/blob/main/.circleci/config.yml#L484
[2] #185

[sssoleileraaa]

can 1.0 entries be removed?

[kushaldas]

can 1.0 entries be removed?

I think we should do it later with a new PR as we mess up, it will difficult to figure out where it broke (if it is the new work or removing old things).

[kushaldas]

Oh, I guess you are talking about removing the old MarkupSafe wheel removal, that can be done.

[kushaldas]

I could not do it via packages/tarballs from current main of those packages, as I will have to remove the signing check and all related checks etc. Can anyone please try to write it in a better way?

[conorsch]

I haven't taken a look at this yet, @kushaldas, but will do so later in the week.

[kushaldas]

@conorsch I tried it in #214 and I think finally I got it properly.
There I am rebuilding the tarball for securedrop-proxy and securedrop-log and disable gpg signature. We can enable gpg signature after the next release of those packages.

[conorsch]

@kushaldas Nice work in #214! I took the liberty of appending yet more partial work, specifically using reprotest to verify that the checksums matched on the .deb files, same as on the .whl files (#211). As part of that work, I pulled in your test-only commits into this PR, and I've also re-broken the tests. As part of final review, let's make sure to drop all the test-only commits.

Based on the new changes, I have a couple immediate observations:

1. The securedrop-export package needs its own PR, since it's still building from the pypi.securedrop.org/ index on this branch
2. reprotest requires that build artifacts be relative to the project directory. That's already true of the whls, but the debs we've been placing in ~/debbuild/packaging/, and I propose we move that logic to ./build/debbuild/packaging/ in this PR. I've tried to make that migration here, but it's not passing in CI yet.
3. Assuming make test passes for you locally, and in CI, it looks like we can remove the following CI checks:

• build-buster-securedrop-client
• build-buster-securedrop-proxy
• build-buster-securedrop-log
• build-buster-securedrop-export (after we have the checksums updated, mentioned above)

I'm suggesting removal of those tests on every PR run, because the reprotest suite effectively verifies that the build passes—and then passes again! If you agree, please take a stab at adjusting the settings. I'd also appreciate your help on getting the new reprotest suite passing in CI. You'll see that I've moved it from a machine to container executor in CI, but that itself isn't enough. See what you can figure out. We're almost there! 🚀

[conorsch]

Took another look at this today. Looks like we're running into a CI-specific problem with setarch, just as we saw in e293606. It appears this problem prevents us from using reprotest to verify the deb package builds, because we can either use Focal VM images, which cause the wheel lookups to fail, since Focal uses python3.8 rather than Buster's 3.7, or we can use a Buster container image, which will fail on setarch. I haven't found a way to tell reprotest to skip the setarch line, so I believe we're blocked on that for now.

I'll work on adjusting the tests to accommodate. Overall, the logic here is very much what we want! Will poke at reprotest in CircleCI containers another day to understand the root cause better.

[emkll]

@conorsch @kushaldas have you tried the "next gen" circle images? They have python buster images that can be used here:
https://circleci.com/docs/2.0/circleci-images/#python

We have recently switched to these for some core CI targets: freedomofpress/securedrop#5720

[conorsch]

No, but worth a shot! Thanks for mentioning.

[conorsch]

The securedrop-export package needs its own PR, since it's still building from the pypi.securedrop.org/ index on this branch

Fixed in this branch. A clarification, though: yes, the build logic was erroneously using pypi.securedrop.org, but the securedrop-export package doesn't have any external dependencies, so that wasn't breaking the build. Therefore no additional PR is required. Updated the docs to remove mention of the pypi.securedrop.org URL entirely, since we plan to remove it in the future.

have you tried the "next gen" circle images?

I tried those, but to no avail. The problem appears to be that in CircleCI, containers are run with reduced privileges, prohibiting the use of the setarch call. Unfortunately reprotest doesn't allow us to disable that call via configuration, so I patched the file only in the CI env: https://github.com/freedomofpress/securedrop-debian-packaging/pull/213/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R306 Not the prettiest solution, but it lets us remove the manual checksum comparison specific to the CircleCI testing logic. The reproducibility checks can now be run locally, in addition to CI, which should help greatly in future debugging sessions.

CI is now passing across the board, so I'm marking ready and proceeding with final review.

[conorsch]

Merged the associated component PRs, then reverted the temporary commits on this branch. See fully passing build here, with temporary commits reverted: https://app.circleci.com/pipelines/github/freedomofpress/securedrop-debian-packaging/885/workflows/3fb0f623-77a7-4b16-9c9b-2497e8866f5c I'll rebase again to drop those temporary commits entirely, confirm passing, then merge.

[conorsch]

After merging all associated PRs, including this one, into the main branches for all repos, I re-ran the tests in CircleCI, and they're showing green across the board:

• securedrop-debian-packaging - https://app.circleci.com/pipelines/github/freedomofpress/securedrop-debian-packaging/887/workflows/bffffdb9-7562-41e5-bd11-029804c1249f
• securedrop-client - https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/548/workflows/67d81d1a-9b59-41e0-b9bf-6fbd1097ab83
• securedrop-log - https://app.circleci.com/pipelines/github/freedomofpress/securedrop-log/62/workflows/50bf7ed8-9039-41a8-84fe-447174c4d87c
• securedrop-proxy - https://app.circleci.com/pipelines/github/freedomofpress/securedrop-proxy/26/workflows/89e74617-2199-4aae-baee-ac354f7f90e6

There may be a few PRs, especially in the securedrop-client repo, that will need to be rebased to keep tests passing. Other than that, no further action required.