"gem install" installs logs and other "noisy", non-reproducible files

[apteryks]

Hi!

When building/installing C extensions via gem install, log files
produced during the build get installed to the GEM_VENDOR location.
This is undesirable because these files are not functional and they
contain non-reproducible information such as temporary compilation
file names or similar.

Would it be possible to not have any log file installed, or have an option to turn it off?

Reproducer, using nokogiri as an example:

$ GEM_VENDOR=output gem install nokogiri-1.13.10.gem --verbose --local --ignore-dependencies --vendor --bindir output/bin

$ find output/extensions/
output/extensions/
output/extensions/x86_64-linux
output/extensions/x86_64-linux/2.7.0
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/mkmf.log
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/nokogiri.so
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlversion.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/SAX.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/entities.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/encoding.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/parser.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/parserInternals.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlerror.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/HTMLparser.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/HTMLtree.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/debugXML.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/tree.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/list.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/hash.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xpath.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xpathInternals.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xpointer.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xinclude.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlIO.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlmemory.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/nanohttp.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/nanoftp.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/uri.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/valid.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xlink.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/catalog.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/threads.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/globals.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/c14n.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlautomata.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlregexp.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlmodule.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlschemas.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/schemasInternals.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlschemastypes.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlstring.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlunicode.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlreader.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/relaxng.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/dict.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/SAX2.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlexports.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlwriter.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/chvalid.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/pattern.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/xmlsave.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxml2/libxml/schematron.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/xsltconfig.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/xslt.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/xsltutils.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/pattern.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/templates.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/variables.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/keys.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/numbersInternals.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/extensions.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/extra.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/functions.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/namespaces.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/imports.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/attributes.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/documents.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/preproc.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/transform.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/security.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/xsltInternals.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/xsltexports.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libxslt/xsltlocale.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libexslt
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libexslt/exslt.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libexslt/exsltexports.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/nokogiri/include/libexslt/exsltconfig.h
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/gem_make.out
output/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/gem.build_complete

The mkmf.log file contains non-deterministic temporary file names, as shown in the following diffoscope:

$ diffoscope /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10{,-check}
--- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10
+++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check
│   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib
├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib
│ │   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby
│ ├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib/ruby
│ │ │   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby
│ │ ├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib/ruby/vendor_ruby
│ │ │ │   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/extensions
│ │ │ ├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib/ruby/vendor_ruby/extensions
│ │ │ │ │   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/extensions/x86_64-linux
│ │ │ │ ├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib/ruby/vendor_ruby/extensions/x86_64-linux
│ │ │ │ │ │   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/extensions/x86_64-linux/2.7.0
│ │ │ │ │ ├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib/ruby/vendor_ruby/extensions/x86_64-linux/2.7.0
│ │ │ │ │ │ │   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10
│ │ │ │ │ │ ├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib/ruby/vendor_ruby/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10
│ │ │ │ │ │ │ │   --- /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/mkmf.log
│ │ │ │ │ │ │ ├── +++ /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10-check/lib/ruby/vendor_ruby/extensions/x86_64-linux/2.7.0/nokogiri-1.13.10/mkmf.log
│ │ │ │ │ │ │ │ @@ -607,15 +607,15 @@
│ │ │ │ │ │ │ │  11: 
│ │ │ │ │ │ │ │  12:   return !!argv[argc];
│ │ │ │ │ │ │ │  13: }
│ │ │ │ │ │ │ │  14: int t(void) { void ((*volatile p)()); p = (void ((*)()))rb_category_warning; return !p; }
│ │ │ │ │ │ │ │  /* end */
│ │ │ │ │ │ │ │  
│ │ │ │ │ │ │ │ "gcc -o conftest
-I/gnu/store/j4z07lyi1ykk8bc68h1p4bpj1il9dn3f-ruby-2.7.4/include/ruby-2.7.0/x86_64-linux
-I/gnu/store/j4z07lyi1ykk8bc68h1p4bpj1il9dn3f-ruby-2.7.4/include/ruby-2.7.0/ruby/backward
-I/gnu/store/j4z07lyi1ykk8bc68h1p4bpj1il9dn3f-ruby-2.7.4/include/ruby-2.7.0
-I. -I/gnu/store/g3y6ifhm0751vgsxv90yipfw6mk189kj-libxml2-2.9.12/include/libxml2
-I/gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/gems/nokogiri-1.13.10/ext/nokogiri/ports/x86_64-linux/libgumbo/1.0.0-nokogiri/include
-g -O2 -fPIC -std=c99 -Wno-declaration-after-statement -g -Winline
-Wmissing-noreturn conftest.c
-L. -L/gnu/store/j4z07lyi1ykk8bc68h1p4bpj1il9dn3f-ruby-2.7.4/lib
-Wl,-rpath,/gnu/store/j4z07lyi1ykk8bc68h1p4bpj1il9dn3f-ruby-2.7.4/lib
-L/gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/gems/nokogiri-1.13.10/ext/nokogiri/ports/x86_64-linux/libgumbo/1.0.0-nokogiri/lib
-Wl,-rpath,/gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/gems/nokogiri-1.13.10/ext/nokogiri/ports/x86_64-linux/libgumbo/1.0.0-nokogiri/lib
-L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -lexslt
-lxslt -lxml2 -lz
/gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/gems/nokogiri-1.13.10/ext/nokogiri/ports/x86_64-linux/libgumbo/1.0.0-nokogiri/lib/libgumbo.a
-Wl,-rpath,/gnu/store/j4z07lyi1ykk8bc68h1p4bpj1il9dn3f-ruby-2.7.4/lib
-L/gnu/store/j4z07lyi1ykk8bc68h1p4bpj1il9dn3f-ruby-2.7.4/lib -lruby
-lexslt -lxslt -lxml2 -lz
/gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/gems/nokogiri-1.13.10/ext/nokogiri/ports/x86_64-linux/libgumbo/1.0.0-nokogiri/lib/libgumbo.a
-lm -lc"
│ │ │ │ │ │ │ │ +ld: /tmp/guix-build-ruby-nokogiri-1.13.10.drv-0/cc6zUuPL.o: in function `t':
│ │ │ │ │ │ │ │ -ld: /tmp/guix-build-ruby-nokogiri-1.13.10.drv-0/ccaC0A0J.o: in function `t':
│ │ │ │ │ │ │ │  /gnu/store/9aba01y3kfdcpxjqg5wx2sbvz4mss5hq-ruby-nokogiri-1.13.10/lib/ruby/vendor_ruby/gems/nokogiri-1.13.10/ext/nokogiri/conftest.c:15: undefined reference to `rb_category_warning'
│ │ │ │ │ │ │ │  collect2: error: ld returned 1 exit status
│ │ │ │ │ │ │ │  checked program was:
│ │ │ │ │ │ │ │  /* begin */
│ │ │ │ │ │ │ │   1: #include "ruby.h"
│ │ │ │ │ │ │ │   2: 
│ │ │ │ │ │ │ │   3: /*top*/

The original issue was reported here: sparklemotion/nokogiri#2755

Thank you!

Note: this was originally miss-filed here: https://bugs.ruby-lang.org/issues/19329

[hsbt]

I'm not sure what you want. Did you want to remove all temporary files like log after gem install?

[apteryks]

It'd be nice if the log files were not installed (copied to the VENDOR_GEM directory) by gem install in the first place, since they are mostly useful to debug the build and are not reproducible.

[apteryks]

Currently distributions with a focus on reproducibility like Guix System are left to delete these files themselves, which feels like a workaround.

[sertonix]

+1

This also causes the output to differ depending on the build machine architecture even when the gem doesn't contain any architecture specific data.

[gemmaro]

Hello, I'm also interested in this topic and would like it to be reproducible. I'm also working on building a ruby-4.0 package on GNU Guix, where reproducible builds matter.

By the way, I suppose this issue might be better filed at bugs.ruby-lang.org rather than this RubyGems repository, since mkmf.log is generated by the mkmf gem, whose implementation lives in lib/mkmf.rb in the Ruby source tree.

[apteryks]

@gemmaro already the case: https://bugs.ruby-lang.org/issues/19329, with the last comment requesting to file it here.

[gemmaro]

@apteryks Yes, I saw it, and I think that it should be re-opened there and this one closed.

[apteryks]

Maybe a PR would work better at this point :-).