Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: add setupHook for deterministic build of .pyc in dependents #25707

Closed
FRidh opened this issue May 11, 2017 · 7 comments
Closed

Python: add setupHook for deterministic build of .pyc in dependents #25707

FRidh opened this issue May 11, 2017 · 7 comments

Comments

@FRidh
Copy link
Member

FRidh commented May 11, 2017

Issue description

The Python interpreters are patched so they can build .pyc bytecode free of certain indeterminism.
When building Python packages we currently set

  # Determinism: The interpreter is patched to write null timestamps when compiling python files.
  # This way python doesn't try to update them when we freeze timestamps in nix store.
  DETERMINISTIC_BUILD=1;
  # Determinism: We fix the hashes of str, bytes and datetime objects.
  PYTHONHASHSEED = 0;

Instead of keeping it just in this function, I propose that we move this to the setupHook of the interpreters.

cc @domenkozar @Mic92 @abbradar @bennofs @lsix and whoever I forgot to whom this may be relevant.

@domenkozar
Copy link
Member

AFAIK this turns off hashdos attack protection

@nh2
Copy link
Contributor

nh2 commented May 11, 2017

Is it not possible to turn (the equivalent of) PYTHONHASHSEED only for interface file generation?

If not, I imagine it shouln't be too hard to add that to upstream python, or patch this functionality in (but I assume a lot of people would like it)?

@FRidh
Copy link
Member Author

FRidh commented May 11, 2017

Randomization is only turned off when building packages. Afterwards, this reproducible .pyc is used but the algorithms should then start doing randomization, not already during build-time.

@FRidh
Copy link
Member Author

FRidh commented May 11, 2017

Relevant issue on Python bug tracker: http://bugs.python.org/issue13703

@FRidh
Copy link
Member Author

FRidh commented May 14, 2017

I asked on Python-Dev and was told that we won't have to worry about the hash DOS attack.
https://mail.python.org/pipermail/python-dev/2017-May/147935.html

@FRidh FRidh self-assigned this May 14, 2017
@FRidh
Copy link
Member Author

FRidh commented May 19, 2017

The setupHook is of course also ran when using Python in a nix-shell and thus also when using nix-shell as shebang. If anyone runs a server from a script running through nix-shell, then that server will be at risk. That will have to be documented.

FRidh added a commit to FRidh/nixpkgs that referenced this issue May 19, 2017
The Python interpreters are patched so they can build .pyc bytecode free
of certain indeterminism.

When building Python packages we currently set

```
compiling python files.
in nix store.
DETERMINISTIC_BUILD=1;
PYTHONHASHSEED = 0;
```

Instead if setting these environment variables in the function that
builds the package, this commit sets the variables instead in the Python
setup hook. That way, whenever Python is included in a derivation, these
variables are set.

See also the issue NixOS#25707.
@FRidh
Copy link
Member Author

FRidh commented May 24, 2017

Closing because this is merged in staging (#25916).

@FRidh FRidh closed this as completed May 24, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants