Implement reproducible gem -> deb conversion #1360
Conversation
|
Pfft. Also found a mac problem locally. will upload fixes once it's happy. |
dankegel
force-pushed
the
reproducible-builds-bis
branch
from
34e5750
to
9559a08
Compare
This is the first step towards supporting bit-for-bit identical output files given identical inputs. Alas, Apple's ar is not too good at reading gnu ar archives, so always use ar_cmd to find ar.
…o support bit-for-bit reproducible gem -> deb conversion In those cases where we can get the release date out of the changelog, use it; otherwise fall back to the value given by SOURCE_DATE_EPOCH aka --source-date-epoch-default. --gem-stagingdir is a bit of a kludge, only needed because no compiler supports https://reproducible-builds.org/specs/build-path-prefix-map/ yet. Could have been global option, but not sure any other package handler invokes compilers? Could hoist it up later. Also: - Defer initializing staging_path so subclasses can sneak in new value - gem: remove build files
dankegel
force-pushed
the
reproducible-builds-bis
branch
from
9559a08
to
2bb5257
Compare
|
For what it's worth, I tried converting 51 gems I use, with commands like ruby -Ilib bin/fpm --debug --gem-stagingdir /tmp/gem.tmp --source-date-epoch-from-changelog --source-date-epoch-default 123 -s gem -t deb -v 2.0.2 rest-client Results: All 51 gems yielded bit-for-bit identical .deb's. About a third of them had usable changelogs: clamp-0.6.5 colorize-0.8.1 domain_name-0.5.20170404 eventmachine-1.2.0.1 Another few had changelogs, but without a recognizable date for this version: addressable-2.5.0 backports-3.6.8 cabin-0.8.1 erubis-2.7.0 netrc-0.11.0 No changelog found: arr-pm-0.0.10 childprocess-0.5.9 crack-0.4.3 daemons-1.2.3 Hmm, if only we could get the attention of the author of arr-pm to add a changelog :-) |
|
|
||
| # Override parent method | ||
| def staging_path(path=nil) | ||
| @gem_staging_path ||= attributes[:gem_stagingdir] || Stud::Temporary.directory("package-#{type}-staging") |
There was a problem hiding this comment.
Long-term I would prefer a simpler user experience for deterministic builds (perhaps a single flag --deterministic-build) that would make these kinds of decisions for the user. For now, this is good :)
There was a problem hiding this comment.
Well, yeah... but long-term this option won't be needed (once compilers obey https://reproducible-builds.org/specs/build-path-prefix-map/ ), so if we're going to have an intermediate workaround, let's make it simple, transparent, and scary so nobody uses it by accident.
| def ar_cmd | ||
| return @@ar_cmd if defined? @@ar_cmd | ||
|
|
||
| @@ar_cmd_deterministic = FALSE |
There was a problem hiding this comment.
TIL Ruby provides a constant FALSE that means false. This should probably be lowercase false (for style) but there is no negative impact to the user, so it's cool.
There was a problem hiding this comment.
Ruby 2.4 warns about this constant, so I'll remove it after merging:
../home/jls/projects/fpm/lib/fpm/util.rb:284: warning: constant ::FALSE is deprecated
/home/jls/projects/fpm/lib/fpm/util.rb:301: warning: constant ::TRUE is deprecated
/home/jls/projects/fpm/lib/fpm/util.rb:241: warning: constant ::FALSE is deprecated
/home/jls/projects/fpm/lib/fpm/util.rb:255: warning: constant ::TRUE is deprecated
|
Did a light review and ran the test suite. Tests passing and overall the code is good. Thank you for writing docs :) |
Partial fix for #1232
Ready for review and merging.
It fetches the source epoch from the changelog for about a third of the gems I tried.
Native gems still have some randomness on Darwin, but I'm not going to worry about that;
docs/source/gem.rst mentions that it's only been verified to produce identical results on Linux.
The following test passes locally on ubuntu 16.04: