Reproducible build is not working. #7808
Assignees
Comments
|
I should mention that all files have Jan 1 1980 timestamp, which is another part of the change. self._builder.build(safe_path, bytecode_compile=False, deterministic_timestamp=True)It looks like |
|
Thank you @ngortheone for opening this. I have been out of the office this week due to a family death, but will return to work tomorrow and will take a look at this. |
|
Should be fixed by #7841. |
|
Actually, the PEXes will still not be reproducible due to We have had that information since first introducing |
|
Opened #7843. Let's move the conversation there. |
Eric-Arellano
added a commit
that referenced
this issue
Jun 4, 2019
#7841) #7734 partially turned on reproducible PEX builds, but did not properly update every call site to avoid including `.pyc` files. As decided there, we do not (at the moment) provide an opt-out to still include `.pyc` files at the expense of reproducible PEXes, because we deem this to be a sensible default. Fixes part of #7808.
Eric-Arellano
added a commit
that referenced
this issue
Jun 4, 2019
…in `./pants binary` (#7843) ### Problem With `./pants binary`, from the start we have injected our own information into `build_properties` to include the run information, which includes things like the `timestamp` and `report_url`. For example, a `PEX-INFO` would include these build properties: ```python "build_properties": {"branch": "no-pyc", "buildroot": "/Users/eric/DocsLocal/code/projects/pants", "class": "CPython", "cmd_line": "pants --cache-ignore binary build-support/bin:shellcheck", "datetime": "Monday Jun 03, 2019 12:27:55", "default_report": "/Users/eric/DocsLocal/code/projects/pants/.pants.d/reports/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824/html/build.html", "id": "pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "machine": "Erics-MacBook-Pro.local", "path": "/Users/eric/DocsLocal/code/projects/pants", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "report_url": "http://localhost:49468/run/pants_run_2019_06_03_12_27_55_249_2fd25ab784514e618dc2f05af46dd824", "revision": "4948b9f87d4cf3856c89e3b456da3ce22c8c93f7", "timestamp": "1559590075.249843", "user": "eric", "version": "1.17.0rc0"} ``` Not only are most of these irrelevant when shipping a PEX, but they also cause any PEX created via `./pants binary` to never be able to be reproducible, meaning that #7734 and #7841 will not fully fix reproducible builds (#7808). ### Solution Default to not including this run information, as it is generally not relevant to built PEXes and we do not anticipate the average user exploding the `.pex` file to inspect this data. However, we do introduce a new option `--binary-py-include-run-information` that allows the original behavior. Because we have included this information in `PEX-INFO` from the start, some users may have come to depend on it and should have the option to keep the behavior. ### Result With #7841 also included, two built PEXes will be byte-for-byte identical, e.g. ```bash $ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 1.pex $ ./pants --cache-ignore binary build-support/bin:shellcheck; mv dist/shellcheck.pex 2.pex $ cmp 1.pex 2.pex ``` [`--cache-ignore` is used to ensure here that a new PEX is being generated each time, rather than relying on the prior cached result.] The `PEX-INFO` will now only include this much saner set of values for `build_properties`: ```python "build_properties": {"class": "CPython", "pex_version": "1.6.7", "platform": "macosx_10_14_x86_64", "version": [3, 6, 8]} ```
|
@Eric-Arellano when can I expect these changes in dev release? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@Eric-Arellano @stuhood
I am using 1.17.0.dev1 (5/22/2019) version
https://www.pantsbuild.org/notes-master.html
[GLOBAL] pants_version: 1.17.0.dev1The release mentions the change I am interested in:
./pants binarycommand #7734) PR Turn on reproducible PEX builds, e.g. for./pants binarycommand #7734But my python_binary pex archives still have
pycfiles, and have different sha every timeI have cleaned all caches and even re-installed pants. Please let me know if any additional changes are required to activate reproducible builds?
The text was updated successfully, but these errors were encountered: