Python: deterministic interpreters #22585

FRidh · 2017-02-09T09:17:09Z

Motivation for this change

Reproducible builds of the interpreters

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-02-09T09:17:11Z

@FRidh, thanks for your PR! By analyzing the history of the files in this pull request, we identified @domenkozar, @edolstra and @peti to be potential reviewers.

FRidh · 2017-02-09T09:30:42Z

Python 2.7 diff #22570 (comment)
Python 3.5 diff https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b

FRidh · 2017-02-09T13:42:29Z

Since 3.2.3 there is a PYTHONHASHSEED env var.

If this variable is not set or set to random, a random value is used to seed the hashes of str, bytes and datetime objects

It also influences sets. When doing a x in {'a', 'b', 'c'} the set is converted to a frozenset. If PYTHONHASHSEED is set to 0, then no hashing is done and the order should be preserved (even though sets have no order).

Unfortunately, the bytecode is still not deterministic.

edanaher · 2017-02-13T03:22:38Z

pkgs/development/interpreters/python/mk-python-derivation.nix

  DETERMINISTIC_BUILD=1;
+  # Determinism: We fix the hashes of str, bytes and datetime objects.
+  PYTHONHASHSEED = 0;


This only applies to the build process, right? I assume that the actual scripts will still run with random hash seeds; but we don't want to accidentally open up NixOS to hash collision attacks

We still need to disable it for nix-shell.

I disagree. What is nix-shell used or designed for? For testing/building a package, or for building environments with certain packages? In my point of view its designed for the former, but its (mis)used a lot for the latter as well. If we are going to make changes for nix-shell then we should also unset SOURCE_DATE_EPOCH and likely a dozen other vars as well which will render it useless for the former use case.

In Nix 1.12 we have nix run which, if I am correct, shouldn't set such env vars (unfortunately it still does).

@FRidh I don't think every user is aware of this, so their needs to be a big warning somewhere. Somebody may use nix-shell --command in their own services/timer so this could make people vulnerable to hash collision attacks.

Also there is the nix-shell -i shebang feature for which the Nix documentation even includes a Python example.

FRidh · 2017-02-19T10:39:14Z

At the end of the installation I now force it to rebuild all bytecode. Both 2.7 and 3.5 are now deterministic.

cc @edanaher could you check regarding your issue?

FRidh · 2017-02-19T10:40:10Z

pkgs/development/interpreters/python/cpython/2.7/default.nix

+        find "$out" -name 'wininst*.exe' | xargs -r rm -f
+
+        # Determinism: rebuild all bytecode
+        # We exclude lib2to3 because that's Python 2 code which fails


here the reason is slightly different...but it includes incorrect syntax

edanaher · 2017-02-19T21:59:05Z

Looks good to me; nvr is nice and fast, and there's only a single EROFS:

[nix-shell:~/nixpkgs]$ strace nvr  2>&1 | grep EROFS
open("/nix/store/33mqlj6pchixmkraylrxm22zia5ji0sc-python3-3.5.2/lib/python3.5/__pycache__/_sysconfigdata.cpython-35.pyc.140390896732008", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0644) = -1 EROFS (Read-only file system)

I still find it a little strange that that file is still attempting to be re-cached, but it's not a significant performance issue. If you're not seeing determinism issues with it, I certainly don't see any problem.

FRidh · 2017-02-20T08:07:25Z

@edanaher I'll have a look at the sysconfigdata issue. For some reason it was removed, and I don't know why.

FRidh · 2017-02-22T15:05:00Z

Versions 2.7, 3.5 and 3.6 are now reproducible. I've applied the patches to 3.4 as well but there's one file left causing problems. I've updated the docs and the release notes. The issue with sysconfigdata I keep for later.

I think this is ready now. What say you?

globin

Apart from the link in the release notes, LGTM.

There is some randomness in the Windows installers. Since we don't need them, we delete them.

- Windows installers are indeterministic and we don't need them. - since Python 3 ensurepip is installed by default. pip is indeteministic and we don't need it. - rebuild bytecode to ensure its deterministic

FRidh · 2017-02-26T13:52:40Z

@globin its not clear to me what you mean but we can fix that later.

vcunat · 2017-03-01T07:22:41Z

@FRidh: this or something else new on staging has broken 3.4 setuptools:

ImportError: No module named 'pyexpat'

Example: http://hydra.nixos.org/build/49482784

FRidh · 2017-03-01T08:10:34Z

Thanks @vcunat , I'll have a look at it later today.

Python: deterministic interpreters (cherry picked from commit 04c41e7)

FRidh · 2017-03-02T12:19:50Z

Python 3.4 is fixed with b588ed9

FRidh · 2017-03-03T06:52:20Z

@vcunat am I correct none of this is in 17.03?

FRidh · 2017-03-03T06:53:06Z

Python 3.6 is fixed with a1f6b8b
Now I'm sure they're all ok.

vcunat · 2017-03-03T07:11:29Z

@FRidh: this PR is in 17.03. I pushed it in response to your comment.

vcunat · 2017-03-03T07:13:15Z

When rereading it, I probably misunderstood you and just cherry-picked the whole merge, but I suppose it will be OK if we pick these fixes as well?

FRidh · 2017-03-03T07:16:34Z

I saw only the merge and none of the commits in 17.03, so that's why I was asking.

Yes, all these commits plus the two fix commits can go in.

vcunat · 2017-03-03T07:33:29Z

For reference, the 17.03 commit: bee7854. I don't know why GitHub doesn't display a mention on this PR – I think it does, normally.

vcunat · 2017-03-03T15:32:30Z

I picked those two commits and tested they fix the builds.

Python: deterministic interpreters (cherry picked from commit 04c41e7)

FRidh added 0.kind: enhancement 2.status: work-in-progress 6.topic: python 6.topic: reproducible builds labels Feb 9, 2017

FRidh mentioned this pull request Feb 9, 2017

Python: determinism of the interpreters and bytecode (pyc) files #22570

Closed

FRidh force-pushed the repr branch from 69d8373 to 3932a0a Compare February 9, 2017 12:12

FRidh changed the base branch from master to staging February 9, 2017 12:12

FRidh force-pushed the repr branch from 3932a0a to 8ff23e8 Compare February 9, 2017 15:30

edanaher reviewed Feb 13, 2017

View reviewed changes

FRidh mentioned this pull request Feb 14, 2017

Python changelog - do not merge #22778

Closed

FRidh force-pushed the repr branch from 8ff23e8 to 2b3212c Compare February 19, 2017 10:38

FRidh commented Feb 19, 2017

View reviewed changes

FRidh added the 9.needs: documentation label Feb 19, 2017

FRidh force-pushed the repr branch from 2b3212c to 23b9993 Compare February 19, 2017 10:58

FRidh self-assigned this Feb 20, 2017

FRidh force-pushed the repr branch 2 times, most recently from d7771d4 to 24f5f22 Compare February 22, 2017 15:01

FRidh changed the title ~~Python: deterministic interpreters - WIP~~ Python: deterministic interpreters Feb 22, 2017

FRidh removed 2.status: work-in-progress 9.needs: documentation labels Feb 22, 2017

FRidh requested a review from globin February 22, 2017 15:06

globin approved these changes Feb 23, 2017

View reviewed changes

Python 2.7: improve determinism

09f6b03

There is some randomness in the Windows installers. Since we don't need them, we delete them.

FRidh added 10 commits February 26, 2017 14:50

Python 3.5: improve determinism

8970a9c

- Windows installers are indeterministic and we don't need them. - since Python 3 ensurepip is installed by default. pip is indeteministic and we don't need it. - rebuild bytecode to ensure its deterministic

Python: mkPythonDerivation: use PYTHONHASHSEED=0

dd3a501

Python 3.5: use system expat and ffi

14a88e7

Python 3.6: improve determinism

1531b5e

Python 3.6: use system expat and ffi

d33f6f4

Python 3.4: improve determinism

1bbf249

Python 3.4: use system expat and ffi

57ded03

Python 3.4: improve determinism

04b7a27

Python: explain deterministic build in docs

37704e9

Python: explain deterministic builds in release notes

f69292d

FRidh force-pushed the repr branch from 24f5f22 to f69292d Compare February 26, 2017 13:51

FRidh merged commit 04c41e7 into NixOS:staging Feb 26, 2017

FRidh mentioned this pull request Mar 1, 2017

Python 3.4 broken (pyexpat) #23325

Closed

vcunat pushed a commit that referenced this pull request Mar 1, 2017

Merge pull request #22585 from FRidh/repr

bee7854

Python: deterministic interpreters (cherry picked from commit 04c41e7)

FRidh mentioned this pull request Mar 17, 2017

Python: improve determinism #21123

Closed

FRidh deleted the repr branch March 17, 2017 17:51

bb010g mentioned this pull request Apr 17, 2019

python3{5,6}: Reintroduce bytecode determinism #59796

Merged

10 tasks

adrianpk added a commit to adrianpk/nixpkgs that referenced this pull request May 31, 2024

Merge pull request NixOS#22585 from FRidh/repr

dc0f616

Python: deterministic interpreters (cherry picked from commit 04c41e7)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: deterministic interpreters #22585

Python: deterministic interpreters #22585

FRidh commented Feb 9, 2017 •

edited

Loading

mention-bot commented Feb 9, 2017

FRidh commented Feb 9, 2017 •

edited

Loading

FRidh commented Feb 9, 2017 •

edited

Loading

edanaher Feb 13, 2017

FRidh Feb 13, 2017

domenkozar Feb 13, 2017

FRidh Feb 13, 2017

Mic92 Feb 13, 2017 •

edited

Loading

dezgeg Feb 19, 2017

FRidh commented Feb 19, 2017 •

edited

Loading

FRidh Feb 19, 2017

edanaher commented Feb 19, 2017

FRidh commented Feb 20, 2017

FRidh commented Feb 22, 2017

globin left a comment

FRidh commented Feb 26, 2017

vcunat commented Mar 1, 2017

FRidh commented Mar 1, 2017

FRidh commented Mar 2, 2017

FRidh commented Mar 3, 2017

FRidh commented Mar 3, 2017

vcunat commented Mar 3, 2017

vcunat commented Mar 3, 2017

FRidh commented Mar 3, 2017

vcunat commented Mar 3, 2017

vcunat commented Mar 3, 2017

Python: deterministic interpreters #22585

Python: deterministic interpreters #22585

Conversation

FRidh commented Feb 9, 2017 • edited Loading

Motivation for this change

Things done

mention-bot commented Feb 9, 2017

FRidh commented Feb 9, 2017 • edited Loading

FRidh commented Feb 9, 2017 • edited Loading

edanaher Feb 13, 2017

Choose a reason for hiding this comment

FRidh Feb 13, 2017

Choose a reason for hiding this comment

domenkozar Feb 13, 2017

Choose a reason for hiding this comment

FRidh Feb 13, 2017

Choose a reason for hiding this comment

Mic92 Feb 13, 2017 • edited Loading

Choose a reason for hiding this comment

dezgeg Feb 19, 2017

Choose a reason for hiding this comment

FRidh commented Feb 19, 2017 • edited Loading

FRidh Feb 19, 2017

Choose a reason for hiding this comment

edanaher commented Feb 19, 2017

FRidh commented Feb 20, 2017

FRidh commented Feb 22, 2017

globin left a comment

Choose a reason for hiding this comment

FRidh commented Feb 26, 2017

vcunat commented Mar 1, 2017

FRidh commented Mar 1, 2017

FRidh commented Mar 2, 2017

FRidh commented Mar 3, 2017

FRidh commented Mar 3, 2017

vcunat commented Mar 3, 2017

vcunat commented Mar 3, 2017

FRidh commented Mar 3, 2017

vcunat commented Mar 3, 2017

vcunat commented Mar 3, 2017

FRidh commented Feb 9, 2017 •

edited

Loading

FRidh commented Feb 9, 2017 •

edited

Loading

FRidh commented Feb 9, 2017 •

edited

Loading

Mic92 Feb 13, 2017 •

edited

Loading

FRidh commented Feb 19, 2017 •

edited

Loading