Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix .pyc files being included to partially get reproducible PEX builds #7841

merged 2 commits into from Jun 4, 2019


None yet
4 participants
Copy link

commented Jun 3, 2019

#7734 partially turned on reproducible PEX builds, but did not properly update every call site to avoid including .pyc files.

As decided there, we do not (at the moment) provide an opt-out to still include .pyc files at the expense of reproducible PEXes, because we deem this to be a sensible default.

Fixes part of #7808.

@Eric-Arellano Eric-Arellano requested review from stuhood, jsirois and blorente Jun 3, 2019

@Eric-Arellano Eric-Arellano changed the title Fix .pyc files being included to get reproducible PEX builds Fix .pyc files being included to partially get reproducible PEX builds Jun 3, 2019


stuhood approved these changes Jun 3, 2019

Fix failing integration test
We no longer pre-compile .pyc files with `./pants test`, so the testproject was unintentionally passing. Instead, we can more explicitly call `./pants run` on the file.

@Eric-Arellano Eric-Arellano requested a review from stuhood Jun 4, 2019


jsirois approved these changes Jun 4, 2019

@Eric-Arellano Eric-Arellano merged commit 9587e91 into pantsbuild:master Jun 4, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:no-pyc branch Jun 4, 2019

Eric-Arellano added a commit that referenced this pull request Jun 4, 2019

No longer default to saving non-deterministic run information to PEX …
…in `./pants binary` (#7843)

### Problem
With `./pants binary`, from the start we have injected our own information into `build_properties` to include the run information, which includes things like the `timestamp` and `report_url`.

For example, a `PEX-INFO` would include these build properties:
"build_properties": {"branch": "no-pyc", "buildroot": "/Users/eric/DocsLocal/code/projects/pants", "class": "CPython", "cmd_line": "pants --cache-ignore binary build-support/bin:shellcheck", "datetime": "Monday Jun 03, 2019 12:27:55", "default_report": "/Users/eric/DocsLocal/code/projects/pants/.pants.d/reports/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824/html/build.html", "id": "pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "machine": "Erics-MacBook-Pro.local", "path": "/Users/eric/DocsLocal/code/projects/pants", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "report_url": "http://localhost:49468/run/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "revision": "4948b9f87d4cf3856c89e3b456da3ce22c8c93f7", "timestamp": "1559590075.249843", "user": "eric", "version": "1.17.0rc0"}

Not only are most of these irrelevant when shipping a PEX, but they also cause any PEX created via `./pants binary` to never be able to be reproducible, meaning that #7734 and #7841 will not fully fix reproducible builds (#7808).

### Solution
Default to not including this run information, as it is generally not relevant to built PEXes and we do not anticipate the average user exploding the `.pex` file to inspect this data.

However, we do introduce a new option `--binary-py-include-run-information` that allows the original behavior. Because we have included this information in `PEX-INFO` from the start, some users may have come to depend on it and should have the option to keep the behavior.

### Result
With #7841 also included, two built PEXes will be byte-for-byte identical, e.g.

$ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 1.pex
$ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 2.pex
$ cmp 1.pex 2.pex

[`--cache-ignore` is used to ensure here that a new PEX is being generated each time, rather than relying on the prior cached result.]

The `PEX-INFO` will now only include this much saner set of values for `build_properties`:

 "build_properties": {"class": "CPython", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "version": [3, 6, 8]}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.