Reproducible build still not entirely repoducible

[ngortheone]

@Eric-Arellano
Subsequent builds produce artifacts with slightly different dependencies

My Pants.ini

[GLOBAL]
pants_version: 1.18.0.dev0

plugins: [
    'pantsbuild.pants.contrib.awslambda-python==%(pants_version)s',
  ]


[python-repos]
repos: [
    "%(buildroot)s/.wheelhouse"
  ]

[python-setup]
interpreter_constraints: ["CPython>=3.7"]

My 3rd party deps

$ cat -p ./python/3rdparty/requirements.txt 
boto3
requests
PyGithub
python-crontab
psycopg2-binary
markdown2
jinja2

Artifacts are almost the same, the only difference is in included dependency. Sometimes it is cp37, and sometimes py3. Which happens randomly, like a coin flip.

python_crontab-2.3.6-cp37-none-any.whl
# OR
python_crontab-2.3.6-py3-none-any.whl

I have only noticed this happening for python_crontab library, but my set of deps is quite small.

P.S. I guess my build is now deterministic, I will definitely get one of the two possible artifacts :)
But that keeps AWS lambda unhappy, since it uses base64(sha256) to track changes, which triggers unnecessary deployments

[Eric-Arellano]

Hello. That is an issue with how Pex (and possibly Pip?) convert an sdist (meaning a bunch of source code) into a wheel.

If you look at PyPi, that particular dependency only provides an sdist https://pypi.org/project/python-crontab/#files. Pex can only work with wheels (aka bdist), so it must convert that sdist into a wheel.

It’s not clear to me why this is sometimes being built as cp37 vs py3. I don't see anything in their setup.py that would trigger this.

I have other things I need to work on for at least this week so can't get to it right away. If you'd like, I'd be happy to support you submitting a patch.

[ngortheone]

Is there something I can do as a workaround on my side to prevent this from happening?

[Eric-Arellano]

Is there something I can do as a workaround on my side to prevent this from happening?

If possible, try an alternative library. In general, both bdist and sdist should be reproducible, per these Pex tests:

https://github.com/pantsbuild/pex/blob/8bd20171f89f1ff1167d55dca38854bef8a60c4f/tests/test_integration.py#L1424-L1430

So, this might be something specific to that library that is causing the issue. It's possible that the test is not comprehensive enough for sdist wheels, but it has never flaked the past month so I think this might be something particular about this sdist.

--

Otherwise, jumpstart the patch to fix this. The first step would be to get the test failing by first replacing that sdist test to use python_crontab. Another thing we may need to do is create and compare more than 2 pexes, e.g. compare the creation of 5 pexes to reduce the risk of flakiness.

[Eric-Arellano]

Closing this because this is a Pex issue, not Pants. All conversation should move to pex-tool/pex#735, and I put a work in progress that gets a consistently failing test at pex-tool/pex#734.

FYI this might be a few weeks before we land this fix back into Pants because we first need to fix the issue in Pex, and then release Pex with the fix, upgrade Pants to use the new version of Pex, and release that new version of Pants.

The best way to speed this all up would be to pull down pex-tool/pex#734, run tox -e py37-integration -- -k test_reproducible_build_sdist -vv, and go from there to start fixing the issues (more than just the wheel name being different, the METADATA file differs too). The error message is fairly descriptive, and it appears to be an issue with dictionary order. We need to figure out where that dictionary is being read and where to set it to be deterministic via sorting or OrderedDictionary (assuming this isn't an issue with upstream pip).