-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support reproducible Python builds #73894
Comments
See https://reproducible-builds.org/ and https://reproducible-builds.org/docs/buy-in/ for why this is a good thing to have in general. Fedora, openSUSE and possibly other Linux distributions package .pyc files as part of their binary rpm packages and they are not trivial to drop [1]. A .pyc header includes the timestamp of the source .py file [1] http://lists.opensuse.org/opensuse-packaging/2017-02/msg00086.html |
--
|
Shouldn't this at least also cover Python 3.7? And should it be officially backported? I would think that if #296 gets accepted for 3.7, then distros that care can cherry pick it back into whatever versions they still support. It probably needn't be officially cherry picked upstream. (FWIW, this doesn't affect the Debian ecosystem since we don't ship pycs in debs.) |
backports are optional. |
I have proposed PEP-552 to address this issue. |
Hey, Allow me to join the discussion here. Context:
Current status:
References:
I wanted to share my [and our] interest in this. If we can help in any way, feel free to ping. I will try to hack/patch some more stuff in the current Python releases to make them fully reproducible [for us], and probably share the results here. Thanks |
PEP-552 has been implemented for 3.7. |
Thank you for the heads-up. I just stumbled over this last night. Will keep an eye for 3.7, and see about 2.7. |
A disagreement has popped up over what the ideal solution is on the PR currently connected to this issue. I'm having the folks involved switch it over to here. IMO I think py_compile can respect SOURCE_DATE_EPOCH and just blindly use it for creating .pyc files. That way builds are reproducible. Yes, it will quite possibly lead to those .pyc files being regenerated the instant Python starts running, but SOURCE_DATE_EPOCH is entirely about builds, not runtimes. Plus .pyc files are just optimizations and so it is not critical they not be regenerated again later. |
So, a couple of things. It seems to me, that properly supporting SOURCE_DATE_EPOCH means using exactly that and nothing else. To that end, I'm not entirely sure why things like --clamp-mtime even exist, as the original timestamp of a source file doesn't seem to have a lot of utility and it is better to be entirely predictable. But I'm not going to argue that, except insomuch as it seems IMHO to fit better for python to just keep things simple and override the timestamp with the value of SOURCE_DATE_EPOCH That being said, I see two problems with python implementing something analogous to --clamp-mtime rather than just --mtime.
Of course, in both those cases, blindly respecting SOURCE_DATE_EPOCH will seemingly break everything for people who use --clamp-mtime instead. I'm not happy with reproducible-builds.org for allowing either one. I don't think python should rely on --mtime users manually overriding the filesystem metadata of the source files outside of py_compile, as that is a hack that I think we'd like to remove if possible... that being said, Arch Linux will, on second thought, not be adversely affected even if py_compile tries to be clever and emulate --clamp-mtime to decide on its own whether to respect SOURCE_DATE_EPOCH. Likewise, I don't really expect people to try to reproduce builds using a future date for SOURCE_DATE_EPOCH. On the other hand, the reproducible builds spec doesn't forbid it AFAICT. But... neither of those mitigations seem "clean" to me, for the reasons stated above. There is something that would solve all these issues, though. From reading the importlib code (I haven't actually tried smoketesting actual imports), it appears that Python 2 accepts any bytecode that is dated at or later than the timestamp of its source .py, while Python 3 requires the timestamps to perfectly match. This seems bizarre to behave differently, especially as until @bmwiedemann mentioned it on the GitHub PR I blindly assumed that Python would not care if your bytecode is somehow dated later than your sources. If the user is playing monkey games with mismatched source and byte code, while backdating the source code to *trick* the interpreter into loading it... let them? They can break their stuff if they want to! On looking through the commit logs, it seems that Python 3 used to do the same, until 61b1425 refactored the general vicinity and modified this behavior without warning. In a commit that seems to be designed to do something else entirely. This really should have been two separate commits, and modifying the import code to more strictly check the timestamp should have come with an explanatory justification. Because I cannot think of a good reason for this behavior, and the commit isn't giving me an opportunity to understand either. As it is, I am completely confused, and have no idea whether this was even supposed to be deliberate. |
As Eli's comments are coming off as negative to/at me, I feel like I have If there is a desire to change the semantics of how timestamps are checked On Sat, Jan 13, 2018, 16:57 Eli Schwartz, <report@bugs.python.org> wrote:
|
I think, there is no single nice and clean solution with time-based .pyc files, but to get a whole distribution to build reproducibly, there are two other ways:
on the side-issue: IMHO checking exact mtimes is the right thing to do, because sometimes users will copy back old .py files and expect mismatching .pyc files to not be used. |
Bernhard's idea of SOURCE_DATE_EPOCH being an implicit envvar to forcibly switch on hash-based .pyc files in py_compile is intriguing. I assume this would force the check_source bit to be set? Or since SOURCE_DATE_EPOCH should only be used in build scenarios would you want UNCHECKED_HASH? As the core dev who seems the most engaged and willing to commit this, I'm willing to make the final decision on this and commit the final PR. I see the options of getting this into 3.7 as the following:
That's it. No clamping, no changing how timestamp-based .pyc files are invalidated, no touching source files, etc. If this is going to make it into Python 3.7 then a decision must be made by Friday, Jan 19, so have your opinions on those two options in before then (and in the case of the hash-based solution, would you expect CHECKED_HASH or UNCHECKED_HASH?). At that point I will make a decision and Bernhard can either update his PR or I can create a new one forked from his(I leave that up to Bernhard based on the decision I'll make on/by Friday). |
On Jan 15, 2018, at 11:31, Brett Cannon <report@bugs.python.org> wrote:
I’d suggest that if SDE is set to an integer, that is used as the timestamp. If it’s set to a special symbol (e.g. ‘hash’) then the hash is used. I’m not volunteering to write the code though. :) |
Since Barry chose an option that wasn't listed, I'm planning on accepting Bernhard's #5200 at some point next week barring any new, unique objections. |
Just merged Bernhard's PR which forces hash-based .pyc files. Thanks to everyone who constructively helped reach this point. |
Hey, Sorry, if I'm a bit late to the party with this. The way I validate whether Python is reproducible is with this link: There is a need to also patch getbuildinfo.c to make Python reproducible. I have opened a PR for this : #5313 I've waited for the periodic build to trigger on that reproducible page. There are still some python3 packages that need patching. Alex |
Any chance we can get the (somewhat related) patch for https://bugs.python.org/issue30693 also merged? |
For what it's worth, in Endless OS we still saw slight variations between builds in the .pyc files, even with all the source files' mtimes set to the epoch (ie. equivalent to setting & supporting SOURCE_DATE_EPOCH, I believe). Looking at the contents of the file suggested it was just reordering of class fields; indeed, we only saw this on Python versions where hash randomization is enabled by default, and disabling hash randomization made the output reproducible. |
Yeah, I also see it with 3.6.4. Otherwise I may have to start digging deep into compilation logic. Looking here: More specifically here: build1: 00007f80:·0100·003e·0200·0000·72b6·0000·0072·b500··...>....r....r.. 72b6 and 72b5 like to swap positions sometimes. |
00007f80:·0100·003e·0200·0000·72b6·0000·0072·b500··...>....r....r.. 3e 02 00 00 00 is frozenset(size=2) So it seems set order changed. (or items in the set is appearance order is changed.) Anyway, I think Python 3.7 can't guarantee "reproducible" compile because marshal uses reference count. |
Hum, maybe there is a misunderstanding on the PEP-552 purpose. I understood that the main point of the PEP-552 is to compare hash(<source code>), rather than checking the .py and .pyc file modification time. It doesn't magically make the PYC file content fully reproducible. Correct me if I misunderstood PEP-552 as well. |
PEP-552 was a necessary but not sufficient step on the road towards fully deterministic pycs. The PEP says: "(Note there are other problems [1] [2] we do not address here that can make pycs non-deterministic.)" where [1] and [2] are basically the issues Inada-san has linked. |
This doesn't seem to necessarily impact distutils, so I'm leaving it open despite PEP-632. |
Hi! I've been working on reproducible builds for python-for-android [1,2,3]. Current issues with .pyc files are:
[1] kivy/python-for-android#2390 |
For me, I find that manually rewriting my py files in C using the C python api made my builds reproducible (I think) as well as faster. But luckily C is not the only option now, with .NET becoming better at it, I have made hacks to use .NET Core in Python for cross platform python extensions that do not need to be rebuilt based on the OS 🥳 (however this is slow because the resulting binary is not crossgen'd for the OS's specific cpu). Another good thing with .NET is that then I could use version checking of python before calling newer python C api functions that might not exist in the version they are using 🎉. Another cool thing I find with the standalone windows version is that the python standard library is pyc compiled and placed in a zip file, it would be nice if all distributions switched to that so that way they the option in the installers "compile the standard library" could be removed as they would already be compiled and in a zip file which could speed up python on all of it's distributions in that way (which I think could help make them reproducible). |
#93317 The bug I encountered was not caused by the timestamp, I have tools that control timestamps to keep timestamps consistent. if i turn off compiling of multithreading, it don't appear again. I think it's caused by these characters, these strings are ”{“ ”{{“ ”}“ ”}“ There may be DA/FA differences in some pyc files in the python standard library. If two pythons with differences in the standard library pyc are used to compile the same py file containing the "{" string, the pyc generated will also have DA/FA differences. This is because some pyc files in the standard library will be automatically loaded when the |
Thank you for reporting. It is other source of undeterministic pyc as you said. Single codepoint strings (and empty string) are shared among interpreter. Most simple solution is interning all singletons in |
Do you have any plans to fix this bug in the near future? |
I will fix it before 3.12 become beta if no one fix it until then. |
I wish 3.12 was 4.0 so that way the 2to3 would get nuked. But most likely 4.0 will land in 2050. |
2to3 is deprecated since Python 3.11 and so can be removed in Python 3.13 (or later): #84540 It's a problem with the parser implementation in lib2to3 which doesn't support Python 3.10 grammar. |
What if the entire python standard libraries was to be rewritten in C? I just thought of this and I think it would help eliminate this issue as well as make the interpreter faster because then it can be further optimized. As to where to place the resulting code in why not place it in the pythonX.Y.dll file/ the equivalent file on non-windows to avoid any extra files as well (pyd files). Alternatively python could build in cython into it’s compile() function under python to compile scripts into pyd’s. However the generated c code would have to be deleted to not waste a ton of disk space. |
This is such an enormous task that it will never happen. Plus it would make CPython harder to maintain, and make it harder to share anything in the stdlib with other implementations. |
Lots of pointer management boilerplate, plus Python operators will inflate into lenghty function calls. |
Using Cython as an optional feature enabled in However, we need a person who is ready to add the lines into |
If compiled with PGO twice, it will also produce inconsistencies, and there are many different places. What's the reason for this |
I don't understand what you are saying. |
I don’t understand why #93317 didn’t get linked here. |
It was, but the UI is not very obvious: #73894 (reference) |
I am a novice on python compilation and trying determininstic pyc. As simplest use case, I cloned git repo with a sample .py file in two places (src1 and src2) inside the same build environment (docker container) and compiled both of them using invalidation_mode=CHECKED_HASH and SOURCE_DATE_EPOCH set. But I can see that hash of corresponding .pyc files are different. (source file inside both src1 and src2 have same hash but different file attributes notably mtime) How can I compile source file in src1 and src2 so that correponding pyc have same hash? I am using python version 3.10.12. My apologies if this is not the right place to ask. |
On Mon Feb 26, 2024 at 5:58 PM CET, compete2cooperate wrote:
Look at patches we, openSUSE, have in our packages. Particularly relevant are I believe |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: