Cleanup object files on gems with extensions

[eloyesp]

I've just noticed that gems with compiled extensions use a lot of disk space, most of it are object files that, as far as I understand, are not needed on runtime.

For example, on sassc-2.4.0 does use (on my linux computer) 114.3 MiB, where extension code is 110.9 MiB but the important files (as far as I understand) are just libsass, with 1,7 MB and libsass.so 3,2MB. That is about 108 MB of extra space that could also make slower deployments, when generating bundles and testing when caching gems on CI, or simply on developers machines that duplicate gems on different environments (per project GEM_HOME).

Here are my current environment details:

$ gem env version 3.0.3

What are your thoughts on this?

[eloyesp]

I've just done find $GEM_HOME -iname '*.o' -print0 | xargs -0 rm for testing, and it seems that nothing broke, and saved about 150 Mb of space on a tiny rails project.

[simi]

I think this is not really RubyGems problem, but this should be handled in compiled extension logic itself on the gem side.

sass/sassc-ruby#200 is this the same problem?

[eloyesp]

I'm not sure, the uploaded gem (and the downloaded gem) is pretty small (it does not include the compiled code), it is compiled by rubygems during the install procedure, I'm not sure about the details of the compilation procedure, but it is called by rubygems.

I think that rubygems could add a cleanup step on that procedure (instead of making every gem out there implement it). May be that is not possible, in that case, that responsibility should be better documented, and may be a hook could be provided for gem developers to implement that (I could not found any documentation about this).

As far as I understand, the only hook that the gem developers can use to cleanup object files is the install make task on the Make file that is bundled in the gem. (or generated by mkmf)

On the other hand, I think that the sassc case just add visibility on the issue, but it does happen with every gem with extensions.

If that cleanup step cannot be implemented by rubygems, there might be possible to:

• Add a list of generated files (and cleanup the rest), as there are already many lists of files on the gemspec file.

• Implement that cleanup on Mkmf and related gems, so gems that use it get that cleaning.

[deivid-rodriguez]

Not sure where the code implementing this should live, but automatically cleaning up generated .o files after installation sounds like a good idea to me.

[simi]

@deivid-rodriguez in theory it could be possible to add another clean at the end here https://github.com/rubygems/rubygems/blob/3e0641a6c8eff4d9a5d26856b1ac7417168b4cc0/lib/rubygems/ext/builder.rb#L42
But the impact of this change is unknown for me.

[eloyesp]

Running make clean might remove also the important files (sassc and sassc.so on our example).

[simi]

@eloyesp but those should be already copied to proper place during install AFAIK.

[retro@retro  ext (master=)]❤ make install
/usr/bin/install -c -m 0755 libsass.so /home/retro/.rubies/ruby-2.5.7/lib/ruby/site_ruby/2.5.0/x86_64-linux/sassc

[immerrr]

Just ran into the same sass issue, and after tracing the installation procedure all the way through gem install ended up here. In my experience, it is generally fine to clean or even distclean the source directories after the binary program/library has been built and installed.

[deivid-rodriguez]

I don't forseee any clear bad consequences from doing this, so I'd say feel free to propose a PR if interested.

[eloyesp]

It seems that calling clean does not break anything, but I've just tested locally with a rails app and modifying that single line. The gain was about 90 Mb.

-    ['clean', '', 'install'].each do |target|
+    ['clean', '', 'install', 'clean'].each do |target|

[immerrr]

Yeah, I have seen a similar gain: yesterday I added a find /usr/local/bundle/gems -name '*.o' -delete to a dockerfile of an application, and the resulting image size went down ~250Mb.