Skip to content

Preview: deterministic build #2281

Open
wants to merge 40 commits into from
@alexanderkjeldaas

This is a set of changes that makes the system_tarball_pc derivation deterministic.

Stdenv bootstrap:

  • The stdenvs created during bootstrap are numbered to make it easier to debug/understand
  • BUG: for the bootstrap stdenvs, binutils was after the bootstrap binaries in path.
  • Deterministic archives are enabled early during bootstrapping of stdenvs.

Stdenv builder:

  • The generic builder depends on libfaketime. libfaketime needs a home. libfaketime is currently distributed through github, that that does not work during bootstrap when SSL is not available. It should be moved to tarballs.nixos.org.
  • Feature: Setting useFakeTime fixes the time during builds. This can break builds, but is easy to manage compared to patching. Additional environment variables are available for configuring libfaketime.
  • Feature: A fake "date" utility is prepended to the path
  • Check: The build directory (/tmp/nix-build-foo-x.y.z) is not allowed to appear in artifacts ($out) to avoid non-deterministic output.

Gcc:

  • Gcc by default now defines DATE and TIME to be (time_t)0.
  • TODO: -frandom-seed is not set. This is relevant for c++ code.
  • PGO is turned off for gcc. Note: there is no consensus on doing this as it reduces compilation performance.

Various changes:

  • The xsltproc utility creates random identifiers. A post-processing stage using perl was added to the nixos manual. This could be generalized.
  • All uses of 'gzip -9' has been replaced with 'gzip -9n'.
  • Added options to cpio for default uid/gid, and default mtime for tar.
  • File lists are sorted before being added to cpio/tar.
@alexanderkjeldaas

Known issues:

  1. The inode changes done in perl seems to be buggy (boot problems), but they might not be needed.
  2. Some patches are not required because of the gcc-wrapper and libfaketime features.
  3. I'm not sure the overridden 'date' binary works correctly.
@shlevy
Official Nix/Nixpkgs/NixOS member
shlevy commented Apr 16, 2014

I think nix uses the equivalent of (time_t) 1 for its file a/m/ctime changes

@domenkozar
Official Nix/Nixpkgs/NixOS member

Wonderful. @alexanderkjeldaas that means most of other PRs can be closed?

@thoughtpolice
Official Nix/Nixpkgs/NixOS member

@vcunat @alexanderkjeldaas Can we get some of these things merged on the pending stdenv branch? I know that we haven't solved the GCC question re: PGO, so we'll have to leave that commit out, but a lot of these changes are not very intrusive, and merging them would reduce burdens later and get us much closer to a deterministic build.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Jun 9, 2014

Ah, I completely forgot this series of work. The currently staged stdenv has been quite tested by me, so I would merge it about the current state (after Hydra verifies gcc on Darwin). I'll look at this afterwards, as I wanted to do another iteration of stdenv stuff (there were some others I missed this time).

@vcunat vcunat added the enhancement label Jun 9, 2014
@alexanderkjeldaas
@vcunat vcunat added the stdenv label Jun 9, 2014
@thoughtpolice
Official Nix/Nixpkgs/NixOS member

@alexanderkjeldaas I'm going to begin merging some of this work into HEAD soon. I'm probably not going to merge everything in one go, so feel free to rebase this when you get a chance. I'll update with what I've pushed upstream.

@alexanderkjeldaas
@alexanderkjeldaas

I've added a minor fix for python 2.7.7 that I forgot to cherry-pick from my internal branch.

@7c6f434c
Official Nix/Nixpkgs/NixOS member

So, what is the status of cherry-picking? Github doesn't easily show this, unfortunately…

Obviously, this will never get directly in master (only in staging) and it gets chery-picked in small pieces.

I actually support reproducibility, although some people seem to like PGO too much…

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Aug 30, 2014

PGO is probably the only questionable thing here, IIRC. I'm planning to really review and test this within the next 10 days.

alexanderkjeldaas added some commits Oct 1, 2013
@alexanderkjeldaas alexanderkjeldaas Make glibc compilation more pure.
Remove datetime from nscd.
1472989
@alexanderkjeldaas alexanderkjeldaas Make perl 5.16 binary deterministic. 2410fa1
@alexanderkjeldaas alexanderkjeldaas Make the linux bootstrap environments more deterministic.
This includes two changes:
1) Fix a bug where the bootstrap-tools is always used instead of binutils
2) Enable strip --enable-deterministic-archives as soon as a new binutils
   is available.
61c55ad
@alexanderkjeldaas alexanderkjeldaas Improve python library determinism.
1) Make the core python libraries deterministic.
2) Make the python libraries created by glib deterministic.
0d6467c
@alexanderkjeldaas alexanderkjeldaas Improve determinism for libgcrypt, libgpg-error, and busybox. 84a9791
@alexanderkjeldaas alexanderkjeldaas Make openssl deterministic. 7f014c7
@alexanderkjeldaas alexanderkjeldaas Make groff and opensp deterministic. 30a92c4
@alexanderkjeldaas alexanderkjeldaas Make __DATE__/__TIME__ deterministic when NIX_ENFORCE_PURITY=1 3dba999
@alexanderkjeldaas alexanderkjeldaas Added a fixed time using libfaketime to stdenv. 31e433b
@alexanderkjeldaas alexanderkjeldaas perl-modules: Do not create perllocal.pod, for determinism. f82b30f
@alexanderkjeldaas alexanderkjeldaas Make libfaketime deterministic.
Also, do not link with rt to avoid duplicate definition error.

This was apparently something others have seen as well:
https://github.com/wolfcw/libfaketime/blob/master/packaging/Linux/Debian/avoid-spurious-lrt.patch
cb8bd05
@alexanderkjeldaas alexanderkjeldaas Remove dates from kernel 3.10.35 2919cee
@alexanderkjeldaas alexanderkjeldaas Add a fake date utility together with setup. c232efa
@alexanderkjeldaas alexanderkjeldaas Fix typo/breakage in setup.sh f15d234
@alexanderkjeldaas alexanderkjeldaas Use real address for __DATE__ and __TIME__. 36bba24
@alexanderkjeldaas alexanderkjeldaas Remove libfaketime from propagatedUserEnvPkgs and propagatedNativeBui…
…ldInputs.
64ef7aa
@alexanderkjeldaas alexanderkjeldaas When useFakeTime, set FAKETIME unless set. 673415e
@alexanderkjeldaas alexanderkjeldaas Add useFakeTime for python, groff, kernel. 139262e
@alexanderkjeldaas alexanderkjeldaas Do not introduce indeterminism when compressing initrd. 67a921d
@alexanderkjeldaas alexanderkjeldaas Fix non-deterministic id-generation for the nixos-manual. 62f0019
@alexanderkjeldaas alexanderkjeldaas Make python 2.7 deterministic.
Unfortunately this kills the wininst feature as I could
not figure out how to get the .exe files deterministic.
dc2fe83
@alexanderkjeldaas alexanderkjeldaas Make apr-util deterministic. 969208b
@alexanderkjeldaas alexanderkjeldaas Make the system tarball deterministic. 4b78a5b
@alexanderkjeldaas alexanderkjeldaas Not patching elf binaries result in non-deterministic builds.
I have no idea why dontPatchELF was set.  If it was not important enough to comment
on it might have been a spurious issue.
15cb459
@alexanderkjeldaas alexanderkjeldaas Fix date in version string.
ntp cannot be safely built with useFakeTime because
the version construction system detects how many
times the version string is constructed, and it can
be constructed more than once when the time is
fixed by libfaketime.
d6eefe6
@alexanderkjeldaas alexanderkjeldaas Make smartmontools deterministic. fd1101a
@alexanderkjeldaas alexanderkjeldaas Set useFakeTime on a set of derivations.
This might not be the best of ideas as this can cause
build problems.
However, it makes the build deterministic for now.
5b81457
@alexanderkjeldaas alexanderkjeldaas Do not set nlink when cleaning cpio archives. 1462666
@alexanderkjeldaas alexanderkjeldaas Create consistent initrd cpio archive.
1) Sort the files
2) Set uid/gid to 0.0
3d28eb3
@alexanderkjeldaas alexanderkjeldaas Disable useFakeTime for smartmontools. 855843b
@alexanderkjeldaas alexanderkjeldaas Eradicate gzip -9 without -n a8e7ddd
@alexanderkjeldaas alexanderkjeldaas Set the linux kernel timestamp properly. f97b73b
@alexanderkjeldaas alexanderkjeldaas Make syslinux deterministic. e8ee36a
@alexanderkjeldaas alexanderkjeldaas Disable profile-guided-optimization (PGO) for gcc.
This only disables it for compiling gcc itself.
PGO results in non-deterministic output because the timing information itself
has small variations.
0ca9e10
@alexanderkjeldaas alexanderkjeldaas Barf on finding the build directory in build artifacts. e52a6a2
@alexanderkjeldaas alexanderkjeldaas Remove build artifacts from e2fsprogs output. 3157dbe
@alexanderkjeldaas alexanderkjeldaas Change some fixed timestamp to != (time_t)0 449193d
@alexanderkjeldaas alexanderkjeldaas Add atomic-ops package. 3823348
@alexanderkjeldaas alexanderkjeldaas Excluding directories makes initrd fail for the kernel. 4156801
@alexanderkjeldaas alexanderkjeldaas python 2.7.7 updates to deterministic builds. 90c307e
@alexanderkjeldaas

Rebased

@vcunat vcunat self-assigned this Sep 13, 2014
@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 13, 2014

Please, don't rebase from now on. I'm going through the commits, also porting them to staging on the way. I'll keep the WIP on https://github.com/vcunat/nixpkgs/commits/staging

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented on 1472989 Sep 13, 2014

Modified by e316672. Explained in the commit.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 13, 2014

BTW, I do welcome any comments on the changes I do to the commits (or on the original commits).

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented on 61c55ad Sep 13, 2014

I don't think this is needed anymore. Our binutils already produce deterministic archives by default (i.e. unless overridden).

Line 95 is a bugfix.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented on 3dba999 Sep 13, 2014

I modified it a bit, see aa119e1.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 13, 2014

Pushed the first few into staging.

@vcunat vcunat referenced this pull request Sep 16, 2014
@alexanderkjeldaas alexanderkjeldaas gcc-wrapper: make __DATE__/__TIME__ deterministic
...when NIX_ENFORCE_PURITY=1.

@vcunat corrected the date according to docs.
https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
In order to handle the spaces well, the extraAfter array had to be
quoted more properly and appended by +=.
aa119e1
@alexanderkjeldaas
Official Nix/Nixpkgs/NixOS member

I don't know how that change relates to the current state in the staging branch. The whole bootstrapping process was refactored by @errge and I haven't yet spent time to understand it. I'll queue this in my TODO list.

I think this pull request just have to be rebased on top of current staging.

I agree with the numbering of the stages, actually I wanted to do something similar (showing in the name of the stage, which stage it is).

As @vcunat already said, the deterministic flag is not needed most probably anymore, so I wouldn't add that "if" unless it's absolutely needed.

I'm currently waiting for stabilization of the current staging on hydra, but after that I plan to submit and get merged a PR for actually rebuilding glibc and gcc one more round, to make sure that we don't even have a libc on the system that has been built by bootstrap binutils. This may help in achieving determinism in stdenv.

Also, when you rebase this pull request on top of current staging, may I ask you to instead of just saying "this is a bugfix", explicitly explain what is the bug and why do you think your proposal is a solution?

@errge Sorry for such a short response, but this is explained both in the commit message and as a comment on line 93..95.

What happens is that bootstrap-tools includes some random non-deterministic tools, like strip. As long as these tools are used, it is irrelevant whether we build a deterministic strip. It won't be used. We need to put the binutils that we actually build in stage-N at the front of the path in stage-(N+1), not the bootstrap-tools version.

FYI, I don't have time right now to rebase, I both need to ingest the current changes, and get some time to look at nixpkgs.

@errge if, as you say, binutils isn't being passed along the stdenvs, then maybe the path order bug is gone as well.

@errge Now I remember a bit more. initialPath is a way to specify a default path during bootstrap. So it is initially populated with the bootstrap tools. As long as this initial path is given preference over the binutils we later build, then we are at the mercy of the bootstrap strip behavior.

So this can be solved in other ways, by moving it to the end of the path in various other parts of the system, or by adding a conditional on whether we have come to an stdenv where binutils has been built.

@errge

And this whole passing of binutils seems not necessary anymore with current staging.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 21, 2014

I'm curious about the motivation to provide custom date command, pass the __TIME__ macros, etc., when libfaketime should kill all these cases. You also make libfaketime opt-in per-package. Is that all because there's (expected to be) some nontrivial performance penalty when using libfaketime?

@7c6f434c
Official Nix/Nixpkgs/NixOS member
@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 21, 2014

@7c6f434c: I don't get you. The faketime is only LD_PRELOADed during build time, not at run time.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 21, 2014

Ah, maybe if you meant that we were building by executables with statically-linked glibc, but AFAIK statically linking glibc is considered bad practice anyway.

@7c6f434c
Official Nix/Nixpkgs/NixOS member
@7c6f434c
Official Nix/Nixpkgs/NixOS member
@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 21, 2014

I can see no better way than patching those packages individually.

@alexanderkjeldaas
@alexanderkjeldaas
@alexanderkjeldaas

I did rebase this on the recent stdenv without too much difficulty. I can update this PR if requested.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Sep 29, 2014

Not needed. I have your original commit list locally and I'm picking from that. Have you (or anyone) encountered a test failure in coreutils? It happens for me after picking some of the remaining changes from this PR, only it's non-deterministic and fails with low probability, so I don't even know if it is introduced by the new commits or not.

Currently I'm delayed by solving fontconfig stuff, because it's long-standing problem that now also blocks some updates from other people (#4232).

@alexanderkjeldaas
@bjornfor

libfaketime can be downloaded over HTTP from its homepage: http://www.code-wizards.com/projects/libfaketime/libfaketime-0.9.6.tar.gz

(I've just pushed a libfaketime nix expression to master (a6c21cc) that uses that URL. I need faketime for making deterministic git clones.)

That is great news. Thanks!

@Nukama
Nukama commented Dec 27, 2014

Can the PGO profile be supplied as an input for the nix build system?

@ktosiek
ktosiek commented Dec 27, 2014

@Nukama that'd need someone to provide a "golden profile". Then you'd have to trust that a binary blob:

  1. is what it says it is
  2. is safe (I'm afraid the GCC code that loads profiles for PGO might assume good input)

Plus, it's stdenv so you'd probably want 2 version of the whole OS: PGO one and paranoid one :-P

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Dec 27, 2014

If you're paranoid, it's enough to check any build with a non-PGO gcc, as the output should certainly be the same up to the hashes, only PGO gcc will be faster.

@alexanderkjeldaas
@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Dec 28, 2014

I/we should re-test if these are needed and sufficient for the current state of stdenv bootstrap; but likely the difference won't be too big. I'll try to recover my memory on progress in this. (Now it's a good time to make bigger changes, soon after finishing the release.)

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Dec 30, 2014

I started with stdenv, but perl is still a problem. One thing was that the already-merged patches weren't carried to 5.20 version, which was easy to fix, but I also found other impurities (hostname and other stuff). Perl seems really determined to collect all sorts of information into the output. I still haven't managed to fix all.

I'm comparing builds from two machines: chrooted NixOS, and non-chrooted Fedora.

@peti
Official Nix/Nixpkgs/NixOS member
peti commented Dec 30, 2014

How about this: we create a switch in ~/.nixpkgs/config.nix that says deterministicBuilds = true|false;. If this setting is disabled -- the default -- then we compile gcc with profile-guided optimization, etc. Those who prefer their builds to be totally reproducible -- even at the cost of performance and/or features -- can enable deterministic builds if they wish.

I'm not sure whether Hydra can build (and distribute) yet another copy of all of Nixpkgs, so maybe the deterministic-build crowd wouldn't be able to get more than, say, an extended stdenv from hydra.nixos.org, but this issue feels minor and probably can be solved over time.

@offlinehacker
@vcunat vcunat added a commit that referenced this pull request Jan 3, 2015
@vcunat vcunat linux kernel determinism: unify timestamp style
Testing showed the linux build is sensitive to /usr/include/ncursesw
unless chrooted (on non-nixos).
On a single chrooted nixos machine, -A linux is binary reproducible.

CC #2281 & @alexanderkjeldaas.
6671aff
@alexanderkjeldaas

Since this is discussed on the dev list, @vcunat you are doing a great job on this, but I wonder if we could get a list of the current pain points.

When I did the initial set of changes, I believed i was pretty close to a deterministic system stdenv, so I feel that we should be able to get there if we just get the latest issues fleshed out. If there was a list of things that still needs to be fixed, I might be able to block out some time to take a crack at it.

Is there a particular set of issues that are currently hard to fix?

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Jan 23, 2015

I wanted to start with stdenv and I stuck with perl. There the configure phase collects many things, and I still haven't caught all (some needed me to use two different build machines and kernels to detect them). My WIP on determinism is on https://github.com/vcunat/nixpkgs/commits/v/determ, re-based on some staging version.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Jan 23, 2015

Maybe it would be worth it to patch the uname utility available through stdenv in general (and similar other impurity sources).

@wmertens
@alexanderkjeldaas

uname should be mostly deterministic since the kernel timestamp is deterministic. The hostname might not be deterministic though. I think PID could be made deterministic if the build system moves towards using one container per build. This also makes it possible to make the hostname deterministic by using the UTS namespace.

proot sounds like a good idea, but I haven't seen too many path problems in these patches. I don't see any documentation for proot indicating that it provides hostname, system-time and more. We have the libfaketime which is supposed to set the time to a fixed value, which is a similar low-level approach.

@alexanderkjeldaas

@vcunat thanks for the link! I think just having determinism on a single machine is a great first step. If we can get that for stdenv mainlined, that would be great.

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Jan 23, 2015

uname -a shows e.g. the kernel version of the build machine.

@wmertens
@copumpkin
Official Nix/Nixpkgs/NixOS member

Maybe it would be worth it to patch the uname utility available through stdenv in general (and similar other impurity sources).

👍

@offlinehacker

I tried building system_tarball_pc with this branch and didn't get deterministic results of tarball, any idea how to reproduce this. Do you have some script that checks which store paths are non deterministic for closure?

@alexanderkjeldaas
@domenkozar
Official Nix/Nixpkgs/NixOS member
@lethalman
Official Nix/Nixpkgs/NixOS member

@offlinehacker if I'm not mistaken the tarball is affected by the files in the working dir (e.g. the result symlink itself).

@offlinehacker

Me and @matejc were working on a tool that would rebuild all the packages and check if hashes get changed, will continue when i will have more time.

@lethalman
Official Nix/Nixpkgs/NixOS member

@offlinehacker you are aware of nix-build --check ?

@copumpkin
Official Nix/Nixpkgs/NixOS member

Any progress on this?

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Apr 6, 2015

Not on my side (apart from what I've already posted/linked here).

@copumpkin
Official Nix/Nixpkgs/NixOS member

Can we merge a smaller part of this, @alexanderkjeldaas? It would be a pity for this work to rot. If not, can someone enumerate what's preventing this from being merged?

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented on 62f0019 Jun 2, 2015

I tried this, but nixos-manual remains non-deterministic for me (I tried picking it atop 6276eba).

There is probably something the my @parts = split(/.../) regexp is missing.

@bjornfor

What's the status of this PR? (I don't know if/how I can help, but I'm definitely interested in deterministic builds.)

@vcunat
Official Nix/Nixpkgs/NixOS member
vcunat commented Oct 11, 2015

I cherry-picked some of the commits (long ago). Some packages remained non-deterministic when I tested them so it wasn't clear to me whether the extra complexity was worth it.

@Mathnerd314

I rebased the patches: https://github.com/NixOS/nixpkgs/compare/master...Mathnerd314:deterministic-patches?expand=1

Skipped:

  • These looked like they were random untested changes:
    • python 2.7.7 updates to deterministic builds.
    • Make python 2.7 deterministic.
    • Make syslinux deterministic.
    • Disable useFakeTime for smartmontools.
    • Set useFakeTime on a set of derivations.
    • Make smartmontools deterministic.
    • Add useFakeTime for python, groff, kernel.
    • perl-modules: Do not create perllocal.pod, for determinism.
    • Set the linux kernel timestamp properly.
  • Not patching elf binaries result in non-deterministic builds.
    • This seemed like it could just be a separate PR
  • Change some fixed timestamp to != (time_t)0
    • unclear why these are needed, I did miss a line though
  • Add a fake date utility together with setup.
    • I think just adding a date thing to buildInputs should work
  • Remove dates from kernel 3.10.35
    • Ancient kernel, I don't care about it
  • Add atomic-ops package.
    • already packaged
  • Use real address for DATE and TIME.
    • this went to gcc_wrapper_old; I think current gcc wrapper patches these just fine
  • these should be fixed by just using a fake date utility:
    • Make openssl deterministic.
    • Improve determinism for libgcrypt, libgpg-error, and busybox.
    • Make perl 5.16 binary deterministic.
    • Fix date in version string.
@domenkozar
Official Nix/Nixpkgs/NixOS member

@Mathnerd314 let's open a PR and merge those?

@copumpkin
Official Nix/Nixpkgs/NixOS member

@Mathnerd314 @domenkozar did you end up creating that PR?

@vcunat vcunat removed their assignment Dec 7, 2015
@copumpkin
Official Nix/Nixpkgs/NixOS member

Gcc by default now defines DATE and TIME to be (time_t)0.

I think @edolstra just recently changed this behavior in some builds which might conflict with this.

@copumpkin
Official Nix/Nixpkgs/NixOS member

81e530a is the commit I'm talking about.

@jagajaga
Official Nix/Nixpkgs/NixOS member
jagajaga commented Mar 4, 2016

Ping all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.