Switch to mini_mime for memory savings #1059

Closed
wants to merge 2 commits into
from

Projects

None yet

6 participants

@SamSaffron
Contributor
SamSaffron commented Dec 14, 2016 edited

The mime types gem (even with the magical columnar store by @jeremyevans) is a memory hog.

In particular on boot, mime/types/columnar will allocate 109k objects and end up with a retained count of 31K objects.

This bloat leaves objects that need to be marked and swept every major GC and bloats ruby processes.

To circumvent mini_mime was created.

It uses the exact same database as the full fledged mime types and is capable of only loading stuff on demand with a practical, safe and bound in-memory cache.

mini_mime will allocate 398 objects on boot and only retain 62 (due to rubygems inefficiency)

In table form:

boot allocations boot retained lookup lookup uncached
mini_mime 398 62 641K/s 33K/s
mime-types 109796 31165 361K/s 361K/s

The performance of mini_mime is pretty good, cached lookups are faster than the mime types gem and uncached lookups are 10x slower. (which is really not at huge issue considering the memory savings)

SamSaffron added some commits Dec 14, 2016
@SamSaffron SamSaffron FEATURE: remove the dependency on the mime types gem in place of mini…
…_mime

Also

- Explicit support for Ruby 1.8 and 1.9 is removed
- Appraisal gem no longer needed as it was testing for mime types gem
8b43e72
@SamSaffron SamSaffron try to fix travis 2986672
@mperham
mperham commented Dec 14, 2016

my hero

@SamSaffron
Contributor

FYI did a similar PR to rest-client

rest-client/rest-client#557

@radar
Contributor
radar commented Dec 14, 2016

This is great! Thank you @samsaffron.

Is there any particular reason why you're removing the older versions of Ruby from the travis config? Seems out of scope for this PR.

@SamSaffron
Contributor
SamSaffron commented Dec 14, 2016 edited

@radar it was a bit fiddly to remove the giant matrix of mime types tests and this seemed like the simplest way to resolve, when I removed stuff and left the rake hack in the Gemfile, it had issues bundling.

supporting 1.8 or 1.9 seems very very bad these days ... 1.9 has no support or security patches for almost 2 years.

ultimately it is your call if to support 1.8 but would appreciate some help figuring out an appropriate Gemfile hack here.

@jeremy jeremy added this to the 2.7.0 milestone Dec 14, 2016
@jeremy
Collaborator
jeremy commented Dec 14, 2016

Squashed with #1060 and merged @ 634d911 🖖

@jeremy jeremy closed this Dec 14, 2016
@jeremy jeremy changed the title from FEATURE: remove the dependency on the mime types use mini_mime to Switch to mini_mime for memory savings Dec 14, 2016
@czj
czj commented Dec 14, 2016

Thanks a lot. Ruby and Rails need this kind of optimisation !

@schneems

Sam you are the mini_* master 😎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment