Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gnupg22: reproducible build #87908

Open
wants to merge 2 commits into
base: staging
from

Conversation

@wamserma
Copy link
Contributor

wamserma commented May 15, 2020

GnuPG has a very peculiar way of fixing the date for the documentation.
It relies on a timestamp in the file doc/defsincdate, which is by default
generated by looking at the timestamp of the latest git commit.
If building from a tarball, this file is empty and mkdefsinc will fall
back to the timestamp of the newest source file, which is doc/defsincdate.
Thus, the build date ends up in the documentation.

Creating and populating doc/defsincdate with the the value of
SOURCE_DATE_EPOCH provided by NixPkgs' stdEnv fixes this.

Motivation for this change

GnuPG build was not fully reproducible according to r13y.com

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@wamserma
Copy link
Contributor Author

wamserma commented May 15, 2020

This replaces #78781

@wamserma wamserma force-pushed the wamserma:gnupg-make-reproducible branch from f6d05a5 to 7bd07c2 May 15, 2020
@wamserma wamserma changed the base branch from master to staging May 15, 2020
@ofborg ofborg bot requested review from fpletz, peti and vrthra May 15, 2020
@wamserma
Copy link
Contributor Author

wamserma commented May 15, 2020

An (untested) alternative might be this patch:

--- doc/mkdefsinc.c	2017-08-28 12:22:54.000000000 +0200
+++ doc/mkdefsinc.c.new	2020-05-15 22:54:13.974749616 +0200
@@ -109,7 +109,7 @@
 
   for (; (file = *files); files++)
     {
-      if (!*file || !strcmp (file, ".") || !strcmp (file, ".."))
+      if (!*file || !strcmp (file, ".") || !strcmp (file, "..") || !strcmp (file, "defsincdate"))
         continue;
       if (stat (file, &sb))
         {
Copy link
Member

emilazy left a comment

Built 7bd07c2 on x86_64-linux; here's the hash of the resulting store path, if anyone wants to compare for reproducibility:

emily@renko ~/s/nixpkgs ((7bd07c2d…))> nix build -f . gnupg22
emily@renko ~/s/nixpkgs ((7bd07c2d…))> ls -l result
lrwxrwxrwx 1 emily emily 56 May 15 23:08 result -> /nix/store/0rzxlgi329cy5fjldmqykqzxijghnbnf-gnupg-2.2.19
emily@renko ~/s/nixpkgs ((7bd07c2d…))> nix hash-path result
sha256-0OrnpALUxJlWUhB7AiFuSAlyORdCdT0xxZRZLX7VLpc=
@wamserma
Copy link
Contributor Author

wamserma commented May 16, 2020

@emilazy Same here. (built on NixOS stable).

@wamserma
Copy link
Contributor Author

wamserma commented May 17, 2020

I also put this upstream for discussion: https://dev.gnupg.org/T4947

@wamserma
Copy link
Contributor Author

wamserma commented May 17, 2020

With some more investigation, this actually caused by

postPatch = ''
sed -i 's,hkps://hkps.pool.sks-keyservers.net,hkps://keys.openpgp.org,g' \
configure doc/dirmngr.texi doc/gnupg.info-1
, which leads to the timestamp in doc\defsincdate being erased, as the modification time of doc\dirmngr.texi becomes more recent and hence the cdoc/defsincdateurrent date leadoc/defsincdateks into the build.
This PR works only, because it updates doc\defsincdate after the patch phase(s) and a simple touch doc/defsincdate will do the trick while keeping the original timestamp provided from upstream, hence updated the PR.
nix hash-path result should now return

sha256-XHquyKgPcDEANtboM/rSyY2pCdYigI+eyHoxeFUmMJw=

for

result -> /nix/store/6xy2vh6z6ssajs0bw6dhxwm676y6dsq1-gnupg-2.2.19
@emilazy
Copy link
Member

emilazy commented May 18, 2020

Reproduced sha256-XHquyKgPcDEANtboM/rSyY2pCdYigI+eyHoxeFUmMJw= with e0c1fad.

@wamserma
Copy link
Contributor Author

wamserma commented May 18, 2020

Reproduced sha256-XHquyKgPcDEANtboM/rSyY2pCdYigI+eyHoxeFUmMJw= with e0c1fad.

Which is correct and doesn't help. :) We got both fooled and simply computed the hash of the symlink. Quick confirmation:

../foo1:
lrwxrwxrwx 1 xxx users 6 18. Mai 20:33 result -> ./test
-rw-r--r-- 1 xxx users 4 18. Mai 20:33 test

../foo2:
lrwxrwxrwx 1 xxx users 6 18. Mai 20:34 result -> ./test
-rw-r--r-- 1 xxx users 4 18. Mai 20:33 test

[/tmp/foo2]$ nix hash-path result
sha256-I/i001eIYWHT088PoPGOW0yXbAbEjLz3rxWwBrbWBmo=

[/tmp/foo2]$ cd ../foo1nix hash-path /nix/store/6xy2vh6z6ssajs0bw6dhxwm676y6dsq1-gnupg-2.2.19
sha256-i7fvywV7zXgifoIgnOnn0PH38LBWjj3X+Xkzvt7+kxM=

[/tmp/foo1]$ nix hash-path result
sha256-I/i001eIYWHT088PoPGOW0yXbAbEjLz3rxWwBrbWBmo=

[/tmp/foo1]$ cat test ../foo2/test
foo
bar

So what actually needs verifying is

nix hash-path /nix/store/6xy2vh6z6ssajs0bw6dhxwm676y6dsq1-gnupg-2.2.19
sha256-i7fvywV7zXgifoIgnOnn0PH38LBWjj3X+Xkzvt7+kxM=

which I can confirm for two builds on different days and on different machines.
@emilazy Can you confirm, too?

@emilazy
Copy link
Member

emilazy commented May 18, 2020

Gah, I specifically worried about whether that might be the case but forgot to check...

Thankfully, the hash of the actual tree seems to match as well:

emily@renko ~/s/nixpkgs (gnupg-make-reproducible)> realpath result
/nix/store/6xy2vh6z6ssajs0bw6dhxwm676y6dsq1-gnupg-2.2.19
emily@renko ~/s/nixpkgs (gnupg-make-reproducible)> nix hash-path (realpath result)
sha256-i7fvywV7zXgifoIgnOnn0PH38LBWjj3X+Xkzvt7+kxM=
wamserma added 2 commits May 15, 2020
GnuPG has a very peculiar way of fixing the date for the documentation.
It relies on a timestamp in the file `doc/defsincdate`, which is by default
generated by looking at the timestamp of the latest git commit.
If building from a tarball, this file is empty and `mkdefsinc` will fall
back to the timestamp of the newest source file, which is `doc/defsincdate`.
Thus, the build date ends up in the documentation.

Creating and populating `doc/defsincdate` with the the value of
`SOURCE_DATE_EPOCH` provided by NixPkgs' stdEnv fixes this.
just touch to keep timestamp from upstream instead of forcing it to
$SOURCE_DATE_EPOCH
@wamserma wamserma force-pushed the wamserma:gnupg-make-reproducible branch from e0c1fad to ecb6edb May 22, 2020
@wamserma
Copy link
Contributor Author

wamserma commented May 22, 2020

@GrahamcOfBorg eval
(rebased to current tip of staging)

@wamserma
Copy link
Contributor Author

wamserma commented Jun 3, 2020

@fpletz @peti Do you think this can be merged?
Upstream has added a patch (see #87908 (comment)) that now uses SOURCE_DATE_EPOCH instead of simply blanking out the timestamp then building from the tarball and patching the documentation, while this PR preserves the timestamp by touching the file with the timestamp as extracted from the tarball. (This prevents having epoch instead of a more realistic time/date in the docs.)
When both are present (e.g. after a version update in nixpkgs), the method in this PR should take precedence unless evaluation of the corresponding make target(s) is forced.

@FRidh FRidh added this to WIP in Staging via automation Jun 4, 2020
@FRidh FRidh added this to the 20.09 milestone Jun 4, 2020
@peti
Copy link
Member

peti commented Jun 7, 2020

My preferrence is to go with the solution used by upstream, to be honest.

@wamserma
Copy link
Contributor Author

wamserma commented Jun 7, 2020

@peti This would require that we set SOURCE_DATE_EPOCH to a value derived from the source.

Currently (nixpkgs/master) we get:

$ head -n 3 /nix/store/rak06773gwr9mcycz8p7mnd71zwj1fzj-gnupg-2.2.19/share/info/gnupg.info
This is gnupg.info, produced by makeinfo version 6.5 from gnupg.texi.

This is the 'The GNU Privacy Guard Manual' (version 2.2.19, May 2020).

Using the upstream patch without further adjustment would result into something like

$ head -n 3 /nix/store/???-gnupg-2.2.19/share/info/gnupg.info
This is gnupg.info, produced by makeinfo version 6.5 from gnupg.texi.

This is the 'The GNU Privacy Guard Manual' (version 2.2.19, January 1970).

So my preference is to either keep the simple touch on the timestamp file included with the sourceode to prevent rebuilding it or extracting its value into SOURCE_DATE_EPOCH in a preBuild hook.

edit to keep context: we patch a path in the source for the info file, which then triggers the make-rule for updating the timestamp file, leaking the build time into the info pages.

@wamserma
Copy link
Contributor Author

wamserma commented Jun 15, 2020

ping @peti

@wamserma
Copy link
Contributor Author

wamserma commented Jul 10, 2020

/marvin opt-in

@marvin-mk2 marvin-mk2 bot added the marvin label Jul 10, 2020
@marvin-mk2
Copy link

marvin-mk2 bot commented Jul 10, 2020

Hi! I'm an experimental bot. My goal is to guide this PR through its stages, hopefully ending with a merge. You can read up on the usage here.

@wamserma
Copy link
Contributor Author

wamserma commented Jul 11, 2020

/status needs_reviewer

@marvin-mk2 marvin-mk2 bot added the needs_reviewer label Jul 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Staging
  
WIP
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.