Increased object allocations when using Mail::Address

[tgxworld]

We noticed an increase in the number of heap_live_slots on our Unicorn web workers after upgrading to 2.7.0 and was able to trace it down to the use of Mail::Address.new("someemail@example.com") in our code base. I wrote a simple script that profiles Mail::Address.new and noticed that 2.7.0 is allocating almost 1.7x more objects than 2.6.6

Script

# gem 'mail', '2.6.6'
# gem 'mail', '2.7.0'
# gem 'mail', '2.7.1.rc1'
$LOAD_PATH.unshift '/home/tgxworld/work/mail/lib'
require 'mail'
require 'memory_profiler'

MemoryProfiler.report do
  100.times do
    Mail::Address.new("test@someemail.org")
  end
end.pretty_print

Mail 2.6.6

Total allocated: 5804364 bytes (36504 objects)
Total retained:  2008527 bytes (1039 objects)

allocated memory by gem
-----------------------------------
   3243022  mail-2.6.6
   2553342  rubygems
      8000  other

Mail 2.7.0

Total allocated: 8434106 bytes (61020 objects)
Total retained:  3452215 bytes (568 objects)

allocated memory by gem
-----------------------------------
   4376254  mail-2.7.0
   4049852  rubygems
      8000  other

[jeremy]

Nice catch @tgxworld, thank you. Let's investigate. /cc @SamSaffron

[tgxworld]

@jeremy I spent some time investigating this and was profiling against master which I noticed was allocating lesser than 2.7.0 and 2.6.6.

Mail - master fbc5d91

Total allocated: 4798444 bytes (38090 objects)
Total retained:  2077915 bytes (87 objects)

allocated memory by gem
-----------------------------------
   2395568  mail/lib
   2394876  rubygems
      8000  other

I did a bisect and found that this commit is required to reduce the number of allocations. I don't really understand that commit enough to know why it would help but I thought it might point you in the right direction.

[jeremy]

Thanks @tgxworld! That's a good start.

[ahorek]

most of allocations come from the first load because parsers are lazily loaded

Mail::Address.new("test@someemail.org")
MemoryProfiler.report do
  100.times do
    Mail::Address.new("test@someemail.org")
  end
end.pretty_print

Total allocated: 228800 bytes (4200 objects)
Total retained:  0 bytes (0 objects)

allocated memory by gem
-----------------------------------
    220800  mail-2.7.0
      8000  other

2.7.0

Total allocated: 9087950 bytes (60874 objects)
Total retained:  3451083 bytes (568 objects)

allocated memory by gem
-----------------------------------
    4165622  mail-2.7.0
    4695696  rubygems

    mail     36.412k (± 3.6%) i/s -    183.732k in   5.053053s

2.6.6

Total allocated: 6206485 bytes (35961 objects)
Total retained:  2006240 bytes (1039 objects)

allocated memory by gem
-----------------------------------
   3244622  mail-2.6.6
   2953863  rubygems

   mail     33.416k (± 5.3%) i/s -    168.672k in   5.062884s

2.7.0 + recompiled ragel with -F1 ( https://www.systutorials.com/docs/linux/man/1-ragel/ ) + minor perf tweaks

Total allocated: 3384536 bytes (23029 objects)
Total retained:  1016545 bytes (634 objects)

allocated memory by gem
-----------------------------------
    1231526  mail-2.7.0
    2158192  rubygems

    mail      6.711k (± 4.8%) i/s -     33.748k in   5.041217s

memory allocation with -F1 option is much lower even then 2.6.6, but it's also 5 times slower. Mail gem use -T1 option:
Generate a faster table driven FSM by expanding action lists in the action execute code.

so it's more performance for a cost of more memory consumption. Well it's only 10MB for lookup tables (it isn't allocated over and over again). I don't think we can improve it any better.

btw. Ragel dropped support for Ruby http://www.colm.net/news/2016/07/31/ragel-target-langs.html

[jeremy]

Apparently Ruby support is coming back in v7 😅

https://twitter.com/ehdtee/status/997963900120322050

[tgxworld]

Closing this as more fruitful discussions are taking place in #1343