Reproducible builds: build_path showing variation #231

conorsch · 2021-03-17T21:29:56Z

Discovered while debugging #228. See the diff from two package builds of securedrop-client, built from the 0.4.1 tag, but built in slightly different locations:

│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │ │   --- /tmp/diffoscope_ee2yspmx/tmpiinoiy6_/0/1050.json
│ │ │ ├── +++ /tmp/diffoscope_ee2yspmx/tmpfc5ilrvz/0/1050.json
│ │ │ │┄ Similarity: 0.0%
│ │ │ │┄ Differences: {
│ │ │ │┄   "replace": {
│ │ │ │┄     "dir_info": {},
│ │ │ │┄     "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │┄   }
│ │ │ │┄ }
│ │ │ │ @@ -1,4 +1,4 @@
│ │ │ │  {
│ │ │ │      "dir_info": {},
│ │ │ │ -    "url": "file:///home/user/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │ +    "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │  }

Depending on the buildroot for the package build, we're seeing artifacts within the deb package change. That's not good. A few questions:

Can we catch this type of variations by enabling +build_path for reprotest in CI?
Have we always had this problem, or did it start occurring recently? (We believe it's new.)
Are there documented or unexplained differences in tooling at work here? For example, are newer versions of pip and/or setuptools being pulled in dynamically? Does the behavior change inside a virtualenv vs using dist-packages?

Let's do some research and see what's what.

full diffoscope output

--- debugging/conor/securedrop-client_0.4.1+buster_all.deb
+++ debugging/mickael/securedrop-client_0.4.1+buster_all.deb
├── file list
│ @@ -1,3 +1,3 @@
│  -rw-r--r--   0        0        0        4 2021-03-17 18:20:12.000000 debian-binary
│ --rw-r--r--   0        0        0    33320 2021-03-17 18:20:12.000000 control.tar.xz
│ --rw-r--r--   0        0        0  7870096 2021-03-17 18:20:12.000000 data.tar.xz
│ +-rw-r--r--   0        0        0    33328 2021-03-17 18:20:12.000000 control.tar.xz
│ +-rw-r--r--   0        0        0  7869464 2021-03-17 18:20:12.000000 data.tar.xz
├── control.tar.xz
│ ├── control.tar
│ │ ├── ./md5sums
│ │ │ ├── ./md5sums
│ │ │ │┄ Files differ
├── data.tar.xz
│ ├── data.tar
│ │ ├── file list
│ │ │ @@ -1044,15 +1044,15 @@
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/
│ │ │  -rw-r--r--   0 root         (0) root         (0)        4 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/INSTALLER
│ │ │  -rw-r--r--   0 root         (0) root         (0)    34520 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/LICENSE
│ │ │  -rw-r--r--   0 root         (0) root         (0)    18867 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/METADATA
│ │ │  -rw-r--r--   0 root         (0) root         (0)    12536 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/RECORD
│ │ │  -rw-r--r--   0 root         (0) root         (0)        0 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/REQUESTED
│ │ │  -rw-r--r--   0 root         (0) root         (0)       92 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/WHEEL
│ │ │ --rw-r--r--   0 root         (0) root         (0)      115 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │ +-rw-r--r--   0 root         (0) root         (0)      119 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │  -rw-r--r--   0 root         (0) root         (0)       57 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/entry_points.txt
│ │ │  -rw-r--r--   0 root         (0) root         (0)       18 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/top_level.txt
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/
│ │ │  -rw-r--r--   0 root         (0) root         (0)        4 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/INSTALLER
│ │ │  -rw-r--r--   0 root         (0) root         (0)    35149 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/LICENSE
│ │ │  -rw-r--r--   0 root         (0) root         (0)    10999 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/METADATA
│ │ │  -rw-r--r--   0 root         (0) root         (0)      787 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/RECORD
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/Mako-1.0.7.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/mako-render,sha256=FkzHpLvY33u9MXaS4pmE_2yRxL6CsLwmAQudiHTnJ00,375
│ │ │ +../../../bin/mako-render,sha256=J_dJvEz-DX0bCdRoDjmoAIuLN1LLIcFa9I0XGVOTX_0,379
│ │ │  Mako-1.0.7.dist-info/AUTHORS,sha256=Io2Vw70mjYS7yFcUuJxhIGiMUQt8FWJuxiiwyUW1WRg,282
│ │ │  Mako-1.0.7.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  Mako-1.0.7.dist-info/LICENSE,sha256=hPb4eYmQI51pe21iYx3EApYtT7OJWVXqkm3OeVa69xs,1217
│ │ │  Mako-1.0.7.dist-info/METADATA,sha256=GjlrwIUAxPMmxGogZ0bavg5adWJ0m7cxPmQ-AiFBh_M,2233
│ │ │  Mako-1.0.7.dist-info/RECORD,,
│ │ │  Mako-1.0.7.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  Mako-1.0.7.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/alembic-1.0.2.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/alembic,sha256=wbvkVmx93k_oUgj13pLFT-Bio7tnA2HxHK7ZVkCVJpg,375
│ │ │ +../../../bin/alembic,sha256=G--41PkEELrFREBOM_8CYfypz5WqdRFUWNaStvYVcm4,379
│ │ │  alembic-1.0.2.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  alembic-1.0.2.dist-info/LICENSE,sha256=hJsWUpYBfNjhGU91S2q9caPyPu1klJdvePG7esIrFAI,1184
│ │ │  alembic-1.0.2.dist-info/METADATA,sha256=ly9Z052_B4p_pxRop_DkrF0P70qePxD5tIu-1rHZivs,6038
│ │ │  alembic-1.0.2.dist-info/RECORD,,
│ │ │  alembic-1.0.2.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  alembic-1.0.2.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│ │ │  alembic-1.0.2.dist-info/entry_points.txt,sha256=jOSnN_2fhU8xzDQ50rdNr425J8kf_exuY8GrAo1daz8,49
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/chardet-3.0.4.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/chardetect,sha256=wlBmZ3qG8Rprbmz6bEZHndD6Zq0l0Y17oei7wj-6FkY,383
│ │ │ +../../../bin/chardetect,sha256=x7wS4T3NKiV9U3gkPqPi1KfZ-GQjhX5ORcMrwTYSgbA,387
│ │ │  chardet-3.0.4.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  chardet-3.0.4.dist-info/LICENSE,sha256=YJXp_6d33SKDn3gBqoRbMcntB_PWv4om3F0t7IzMDvM,26432
│ │ │  chardet-3.0.4.dist-info/METADATA,sha256=o6XNN41EUioeDnklH1-8haOSjI60AkAaI823ANFkOM4,3304
│ │ │  chardet-3.0.4.dist-info/RECORD,,
│ │ │  chardet-3.0.4.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  chardet-3.0.4.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│ │ │  chardet-3.0.4.dist-info/entry_points.txt,sha256=fAMmhu5eJ-zAJ-smfqQwRClQ3-nozOCmvJ6-E8lgGJo,60
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/pip-21.0.1.dist-info/RECORD
│ │ │ @@ -1,10 +1,10 @@
│ │ │ -../../../bin/pip,sha256=Ot1LQ47VGHnVx424wHTA0HJPDJKDg7g_Enpt_qV3hA0,387
│ │ │ -../../../bin/pip3,sha256=Ot1LQ47VGHnVx424wHTA0HJPDJKDg7g_Enpt_qV3hA0,387
│ │ │ -../../../bin/pip3.7,sha256=Ot1LQ47VGHnVx424wHTA0HJPDJKDg7g_Enpt_qV3hA0,387
│ │ │ +../../../bin/pip,sha256=ISHOtLHBOkdso4f5K5O6sgYKb1Y2GQOsibXK4co4Emw,391
│ │ │ +../../../bin/pip3,sha256=ISHOtLHBOkdso4f5K5O6sgYKb1Y2GQOsibXK4co4Emw,391
│ │ │ +../../../bin/pip3.7,sha256=ISHOtLHBOkdso4f5K5O6sgYKb1Y2GQOsibXK4co4Emw,391
│ │ │  pip-21.0.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  pip-21.0.1.dist-info/LICENSE.txt,sha256=ejlw8iXn2TntLdOpADqlISSc1qhJJgiYAKMZmq713Gk,1110
│ │ │  pip-21.0.1.dist-info/METADATA,sha256=a6mCPyb1qd3cdVI5OorlrDhSN3HHYiN8feJrxmL4QgY,4168
│ │ │  pip-21.0.1.dist-info/RECORD,,
│ │ │  pip-21.0.1.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
│ │ │  pip-21.0.1.dist-info/entry_points.txt,sha256=5ExSa1s54zSPNA_1epJn5SX06786S8k5YHwskMvVYzw,125
│ │ │  pip-21.0.1.dist-info/top_level.txt,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/RECORD
│ │ │ @@ -1,15 +1,15 @@
│ │ │ -../../../bin/sd-client,sha256=A4hlyqylu8nTK6zfXDbsIF1XU_lFGbhOzJXF4gq2fnA,380
│ │ │ +../../../bin/sd-client,sha256=mqe-XAEPhq3yYJIkernAvpAcLjsjooNpliiVeeG5Eq0,384
│ │ │  securedrop_client-0.4.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  securedrop_client-0.4.1.dist-info/LICENSE,sha256=dql8h4yceoMhuzlcK0TT_i-NgTFNIZsgE47Q4t3dUYI,34520
│ │ │  securedrop_client-0.4.1.dist-info/METADATA,sha256=9v90HnwVWyOt3Q5jEMxY2FUJH9za1CUeMuXGOmalhlU,18867
│ │ │  securedrop_client-0.4.1.dist-info/RECORD,,
│ │ │  securedrop_client-0.4.1.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  securedrop_client-0.4.1.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
│ │ │ -securedrop_client-0.4.1.dist-info/direct_url.json,sha256=niQE_MymeZaFP2DXBdybmQCZ7xGJQ0Lmt3Q_tf1rGcg,115
│ │ │ +securedrop_client-0.4.1.dist-info/direct_url.json,sha256=XJy52Q1fgkeMNe_Dj6D5h-54lG8zwiELec8kUsgdRIo,119
│ │ │  securedrop_client-0.4.1.dist-info/entry_points.txt,sha256=Ky6pbwDDvJfpk4vu3wLB8MB9eSCxa_fzkAD898WY2fE,57
│ │ │  securedrop_client-0.4.1.dist-info/top_level.txt,sha256=-kikgNWO_1DaWx2DRA8yo8Ibcpu5A2Epjx49jAxmF7A,18
│ │ │  securedrop_client/__init__.py,sha256=pMtTmSUht-XtbR_7Doz6bsQqopJJd8rZ8I8zy2HwwoA,22
│ │ │  securedrop_client/__main__.py,sha256=K98BO7GszyT6TPeQmNevXJtJh8StR-HG0rr8GnOIaSc,28
│ │ │  securedrop_client/api_jobs/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  securedrop_client/api_jobs/base.py,sha256=Wg7NsH7eBo7dgcQL-r7Sfwc2PFuCs0e2z5uW_uzY9CM,3923
│ │ │  securedrop_client/api_jobs/downloads.py,sha256=siO9ANtRmx2qTLWWvnH7kUKiDq7UZmwnUxcseABJ2pM,14352
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │ │   --- /tmp/diffoscope_9q8otjqv/tmpvwaebm61/0/1050.json
│ │ │ ├── +++ /tmp/diffoscope_9q8otjqv/tmpdo8n3v94/0/1050.json
│ │ │ │┄ Similarity: 0.0%
│ │ │ │┄ Differences: {
│ │ │ │┄   "replace": {
│ │ │ │┄     "dir_info": {},
│ │ │ │┄     "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │┄   }
│ │ │ │┄ }
│ │ │ │ @@ -1,4 +1,4 @@
│ │ │ │  {
│ │ │ │      "dir_info": {},
│ │ │ │ -    "url": "file:///home/user/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │ +    "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │  }
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/wheel-0.36.2.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/wheel,sha256=J_gtOIynaYEwTx71xxcMqAgMWxwI15CAex3b2zBN5XU,374
│ │ │ +../../../bin/wheel,sha256=spl-_M0tfI2y45a1gtPJi1O8MZmfEflLBpcTg6czOeo,378
│ │ │  wheel-0.36.2.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  wheel-0.36.2.dist-info/LICENSE.txt,sha256=zKniDGrx_Pv2lAjzd3aShsvuvN7TNhAMm0o_NfvmNeQ,1125
│ │ │  wheel-0.36.2.dist-info/METADATA,sha256=w96pAXMHZWzFy-dwqGZqXbxu5Hup3cb3OHujhCuWfDs,2277
│ │ │  wheel-0.36.2.dist-info/RECORD,,
│ │ │  wheel-0.36.2.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│ │ │  wheel-0.36.2.dist-info/entry_points.txt,sha256=N8HbYFST3yrNQYeB2wXWBEPUhFsEtKNRPaCFGJPyqyc,108
│ │ │  wheel-0.36.2.dist-info/top_level.txt,sha256=HxSBIbgEstMPe4eFawhA66Mq-QYHMopXVoAncfjb_1c,6

The text was updated successfully, but these errors were encountered:

We were already checking that building a wheel twice yields the same checksum both times, but we weren't confirming that the checksum matched what's already committed to version control. Let's do that. Still leaving the 'reprotest' check enabled because it supports multiple methods for checking reproducible, which should help us shake out problems like #231.

conorsch · 2021-04-08T00:40:07Z

Dug into this a bit. The build path is embedded inside direct_url.json, as shown in the diffoscope output above. That's following the PEP 610 spec, which appears to be geared toward VCS URLs, not local filepaths. I don't see any reason for us to include that in our debian packages, since it's not required information for the packages to work correctly once installed.

Snipped out that file removes the specific variation reported above, but has a side-effect of altering the checksumming in the RECORD files, according to PEP 376 and PEP 427. Similarly, I don't see any value in shipping these files, given our use of Debian packages for installing Python dependencies. From PEP 427:

RECORD is a list of (almost) all the files in the wheel and their secure hashes. Unlike PEP 376, every file except RECORD, which cannot contain a hash of itself, must include its hash. The hash algorithm must be sha256 or better; specifically, md5 and sha1 are not permitted, as signed wheel files rely on the strong hashes in RECORD to validate the integrity of the archive.

@kushaldas It seems to me we can snip these files out of the Debian package, via the strip-nondeterminism step, to get better reproducibility. Do you see any problems with that?

In #231 we noticed that the filesystem path used to build the debian packages was breaking reproducibility. That's due to the filepath being recorded inside `direct_url.json`, as stipulated by PEP610. Also relevant is PEP427, which describes the `RECORD` file for wheels. Here we remove that file, as well, to ensure full reproducibility regardless of path.

kushaldas · 2021-04-08T16:15:42Z

@kushaldas It seems to me we can snip these files out of the Debian package, via the strip-nondeterminism step, to get better reproducibility. Do you see any problems with that?

I am okay with removing them, I still asked the upstream to get some details (if any).

In #231 we noticed that the filesystem path used to build the debian packages was breaking reproducibility. That's due to the filepath being recorded inside `direct_url.json`, as stipulated by PEP610. Also relevant is PEP427, which describes the `RECORD` file for wheels. Here we remove that file, as well, to ensure full reproducibility regardless of path.

We were already checking that building a wheel twice yields the same checksum both times, but we weren't confirming that the checksum matched what's already committed to version control. Let's do that. Still leaving the 'reprotest' check enabled because it supports multiple methods for checking reproducible, which should help us shake out problems like #231.

conorsch · 2021-04-09T23:05:09Z

Resolved by #244.

We were already checking that building a wheel twice yields the same checksum both times, but we weren't confirming that the checksum matched what's already committed to version control. Let's do that. Still leaving the 'reprotest' check enabled because it supports multiple methods for checking reproducible, which should help us shake out problems like #231.

emkll added this to Near Term - SD Workstation in SecureDrop Team Board Mar 26, 2021

emkll mentioned this issue Mar 29, 2021

pip-21.0.1 is breaking reproducible wheels #218

Closed

sssoleileraaa mentioned this issue Apr 7, 2021

CI should verify a new wheel in localwheels by rebuilding #241

Closed

conorsch mentioned this issue Apr 7, 2021

Tests wheel reproducibility against version control #243

Merged

conorsch mentioned this issue Apr 8, 2021

Adds build_path checks for deb pkg reprotest #244

Merged

conorsch closed this as completed Apr 9, 2021

SecureDrop Team Board automation moved this from Near Term - SD Workstation to Done Apr 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reproducible builds: build_path showing variation #231

Reproducible builds: build_path showing variation #231

conorsch commented Mar 17, 2021

conorsch commented Apr 8, 2021

kushaldas commented Apr 8, 2021

conorsch commented Apr 9, 2021

Reproducible builds: build_path showing variation #231

Reproducible builds: build_path showing variation #231

Comments

conorsch commented Mar 17, 2021

conorsch commented Apr 8, 2021

kushaldas commented Apr 8, 2021

conorsch commented Apr 9, 2021