Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn on reproducible PEX builds, e.g. for `./pants binary` command #7734

Merged
merged 1 commit into from May 16, 2019

Conversation

Projects
None yet
3 participants
@Eric-Arellano
Copy link
Contributor

commented May 15, 2019

Pex 1.6.7 added support for reproducible Pex builds (pantsbuild/pex#716), meaning that the output of pex -o 1.pex will be byte for byte identical to pex -o 2.pex. This specifically means that the PEX will use a deterministic timestamp of January 1, 1980 and that the built PEX will not include .pyc, which is inherently non-reproducible.

In Pex 1.7.0, Pex will default to this reproducible behavior. Originally we were going to wait for that to land to avoid making any changes to Pex. However, a user has a strong desire for reproducible PEX builds so we are turning on the behavior earlier.

Note that we do not introduce any flags to turn off reproducible builds, because we expect that all users will want this behavior. If we get feedback that users do not, e.g. that they want to include .pyc files, then we will add options to turn this behavior off.

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2019

cc @ngortheone

@ngortheone

This comment has been minimized.

Copy link

commented May 15, 2019

@Eric-Arellano Thanks.
When this is merged, do I just need to update pants to the latest version?

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2019

You're welcome!

When this is merged, do I just need to update pants to the latest version?

We do a weekly dev release, usually on Friday or Monday. Once that happens, if you are using the ./pants script that we provide at https://pantsbuild.github.io/setup/pants, then you'll simply change your pants_version in pants.ini to the new version.

To get notified when the release happens, you can join the google group here https://groups.google.com/forum/#!forum/pants-devel.

@stuhood
Copy link
Member

left a comment

Thanks.

@stuhood stuhood merged commit 46c22fa into pantsbuild:master May 16, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:reproducible-pexes branch May 16, 2019

Eric-Arellano added a commit that referenced this pull request Jun 4, 2019

Fix .pyc files being included to partially get reproducible PEX builds (
#7841)

#7734 partially turned on reproducible PEX builds, but did not properly update every call site to avoid including `.pyc` files.

As decided there, we do not (at the moment) provide an opt-out to still include `.pyc` files at the expense of reproducible PEXes, because we deem this to be a sensible default.

Fixes part of #7808.

Eric-Arellano added a commit that referenced this pull request Jun 4, 2019

No longer default to saving non-deterministic run information to PEX …
…in `./pants binary` (#7843)

### Problem
With `./pants binary`, from the start we have injected our own information into `build_properties` to include the run information, which includes things like the `timestamp` and `report_url`.

For example, a `PEX-INFO` would include these build properties:
 
```python
"build_properties": {"branch": "no-pyc", "buildroot": "/Users/eric/DocsLocal/code/projects/pants", "class": "CPython", "cmd_line": "pants --cache-ignore binary build-support/bin:shellcheck", "datetime": "Monday Jun 03, 2019 12:27:55", "default_report": "/Users/eric/DocsLocal/code/projects/pants/.pants.d/reports/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824/html/build.html", "id": "pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "machine": "Erics-MacBook-Pro.local", "path": "/Users/eric/DocsLocal/code/projects/pants", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "report_url": "http://localhost:49468/run/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "revision": "4948b9f87d4cf3856c89e3b456da3ce22c8c93f7", "timestamp": "1559590075.249843", "user": "eric", "version": "1.17.0rc0"}
```

Not only are most of these irrelevant when shipping a PEX, but they also cause any PEX created via `./pants binary` to never be able to be reproducible, meaning that #7734 and #7841 will not fully fix reproducible builds (#7808).

### Solution
Default to not including this run information, as it is generally not relevant to built PEXes and we do not anticipate the average user exploding the `.pex` file to inspect this data.

However, we do introduce a new option `--binary-py-include-run-information` that allows the original behavior. Because we have included this information in `PEX-INFO` from the start, some users may have come to depend on it and should have the option to keep the behavior.

### Result
With #7841 also included, two built PEXes will be byte-for-byte identical, e.g.

```bash
$ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 1.pex
$ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 2.pex
$ cmp 1.pex 2.pex
```

[`--cache-ignore` is used to ensure here that a new PEX is being generated each time, rather than relying on the prior cached result.]

The `PEX-INFO` will now only include this much saner set of values for `build_properties`:

```python
 "build_properties": {"class": "CPython", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "version": [3, 6, 8]}
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.