Rebuild only the wheels which are already not present locally

[kushaldas]

We should include another step before building the wheels locally, that will insure that we only build the wheels which are not build locally.

[kushaldas]

reason

The sha256sum of the binary wheels are different in every rebuild. For example, two different directories after building the wheels in my Debian system.

$ sha256sum localwheels/*.whl
b6e23519f978c7e74718bcf1f22883d07d93623f9bc4283e02b09e77989912bf  localwheels/asn1crypto-0.24.0-py3-none-any.whl
77f65e2a815598c612f74ce11b603bde3ed046f3c954fd9a69688fa668c8ee9b  localwheels/certifi-2018.8.24-py2.py3-none-any.whl
92dfd65ebf5f0ed1db1dd5ff80dedfe317c2f5ba2eb71146e5ce76310bff7fb0  localwheels/cffi-1.11.5-cp35-cp35m-linux_x86_64.whl
e6f7688eec34efeaab34984f60c4548ffce7d8c5d47b29eb03e6967011f33e26  localwheels/chardet-3.0.4-py2.py3-none-any.whl
5f8c5ad6ad3b96f77e78951d8d9db587491bdd9ac293e4f6c07aa7e9a0641cb7  localwheels/cryptography-2.3.1-cp35-cp35m-linux_x86_64.whl
807e69e147502cd6308ce4290a76e93c2f46f1420c86769e7a596de80ee44a21  localwheels/idna-2.7-py2.py3-none-any.whl
3ab9efb840359e0972bb6ddeab5e59e2d4c922bd555dd9e5122305a3e919b967  localwheels/pycparser-2.19-py2.py3-none-any.whl
722e6fea11ff6defd7813810aaf4f0fcdc64dc3d7cc57324a3aca0ad03ea4dc4  localwheels/requests-2.19.1-py2.py3-none-any.whl
65bc4e2e036b91807731e65b54f87a72d64097194403ca8b36af285c543ae10b  localwheels/six-1.11.0-py2.py3-none-any.whl
69766baeed9923b762bd48f6f951aa5f36d9b14aea5884dbeccabf9cb6794d5a  localwheels/urllib3-1.23-py2.py3-none-any.whl

$ sha256sum ../who/localwheels/*.whl
5243fc86fccafcce90da72d3de70fe9349c1123ee5b41c0a556da474e3665345  ../who/localwheels/asn1crypto-0.24.0-py3-none-any.whl
4371bf809103e6ac12bdbf8b8aa7efe2d0919c15a372108abd947c533d3579dd  ../who/localwheels/certifi-2018.8.24-py2.py3-none-any.whl
5e877882a2455e713faad0fc81f716793dbb4fdc844836d04fde771467072752  ../who/localwheels/cffi-1.11.5-cp35-cp35m-linux_x86_64.whl
304d18679149c469bcbc4e0d55d484914d59aa8b61cc8a55b6ecf7146d3bb50d  ../who/localwheels/chardet-3.0.4-py2.py3-none-any.whl
88eec9189d4152b987d279d173a413b05a60fdec5b4a0887fa275e35540016bc  ../who/localwheels/cryptography-2.3.1-cp35-cp35m-linux_x86_64.whl
2788ddf2800cbf57838e528282b96fd08c4cd819dcc740e5576da67d17f781fa  ../who/localwheels/idna-2.7-py2.py3-none-any.whl
ea2a5dacad316cd846d3d4d1a14960a6f287280dd75dd9f1135f81246a39e4a4  ../who/localwheels/pycparser-2.19-py2.py3-none-any.whl
d98dceb375fad80aaec4ff3ce43f1e98a96ae8500f31a1468ae24a32094cf563  ../who/localwheels/requests-2.19.1-py2.py3-none-any.whl
3fb01202ef7d40ad868794a4ae82829648df33c57221c2a9eae745154046f361  ../who/localwheels/six-1.11.0-py2.py3-none-any.whl
d0100fd09abefb0a97517d26e3f4f3952a73d92935c88e1eeccd3ea95a1bf782  ../who/localwheels/urllib3-1.23-py2.py3-none-any.whl

Solution

Instead of manually firing up the pip3 command, we can use a tool which will first verify the existence of the wheel and then only build the wheel if missing. We can have it in the computepipfilehash toolset so that it is available to the developers/package maintainers.

[kushaldas]

Corresponding upstream issue: pypa/wheel#248

[kushaldas]

Even with fixed SOURCE_DATE_EPOCH, the native binary wheels have different sha256sums. For example, cryptography module.

[eloquence]

@kushaldas FYI, looks like we have a couple of upstream responses in pypa/wheel#248

[kushaldas]

I have commented on the upstream issue. Even with latest wheel 0.32.2 package, the binary files have different signature. https://try.diffoscope.org/rhsyskvghvgt.html shows the difference in one such binary file build. I am along talking with the upstream for the reproducible builds. No direct solution other than what we have right now.

[kushaldas]

I am closing this issue for now as we can continue using our documented step.