Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No longer default to saving non-deterministic run information to PEX in `./pants binary` #7843

Merged

Conversation

Projects
None yet
3 participants
@Eric-Arellano
Copy link
Contributor

commented Jun 3, 2019

Problem

With ./pants binary, from the start we have injected our own information into build_properties to include the run information, which includes things like the timestamp and report_url.

For example, a PEX-INFO would include these build properties:

"build_properties": {"branch": "no-pyc", "buildroot": "/Users/eric/DocsLocal/code/projects/pants", "class": "CPython", "cmd_line": "pants --cache-ignore binary build-support/bin:shellcheck", "datetime": "Monday Jun 03, 2019 12:27:55", "default_report": "/Users/eric/DocsLocal/code/projects/pants/.pants.d/reports/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824/html/build.html", "id": "pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "machine": "Erics-MacBook-Pro.local", "path": "/Users/eric/DocsLocal/code/projects/pants", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "report_url": "http://localhost:49468/run/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "revision": "4948b9f87d4cf3856c89e3b456da3ce22c8c93f7", "timestamp": "1559590075.249843", "user": "eric", "version": "1.17.0rc0"}

Not only are most of these irrelevant when shipping a PEX, but they also cause any PEX created via ./pants binary to never be able to be reproducible, meaning that #7734 and #7841 will not fully fix reproducible builds (#7808).

Solution

Default to not including this run information, as it is generally not relevant to built PEXes and we do not anticipate the average user exploding the .pex file to inspect this data.

However, we do introduce a new option --binary-py-include-run-information that allows the original behavior. Because we have included this information in PEX-INFO from the start, some users may have come to depend on it and should have the option to keep the behavior.

Result

With #7841 also included, two built PEXes will be byte-for-byte identical, e.g.

$ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 1.pex
$ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 2.pex
$ cmp 1.pex 2.pex

[--cache-ignore is used to ensure here that a new PEX is being generated each time, rather than relying on the prior cached result.]

The PEX-INFO will now only include this much saner set of values for build_properties:

 "build_properties": {"class": "CPython", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "version": [3, 6, 8]}
@ngortheone

This comment has been minimized.

Copy link

commented Jun 4, 2019

@Eric-Arellano
Can you specify where exacltly --cache-ignore option should be passed? pants help does not show such option

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Jun 4, 2019

Can you specify where exacltly --cache-ignore option should be passed?

@ngortheone updated the PR description with better steps to reproduce that PEXes are now reproducible (with #7841 merged as well).

@stuhood

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

@Eric-Arellano
Can you specify where exacltly --cache-ignore option should be passed? pants help does not show such option

Depending on where it is specified --cache-ignore is shorthand for either ignore caching in all cases, or for the specific task you place it next to. See: https://www.pantsbuild.org/options.html#command-line-flags

But you can see help that includes the cache options by using ./pants help-advanced $task or ./pants help-advanced cache. Alternatively, can see ./pants options to see "all the options".

@ngortheone : But note that --cache-ignore is not part of the solution of the problem: rather, a way to validate the solution.

@stuhood

stuhood approved these changes Jun 4, 2019

Copy link
Member

left a comment

Thanks!

@Eric-Arellano Eric-Arellano merged commit d304be6 into pantsbuild:master Jun 4, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:pex-runtime-information branch Jun 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.