address_lists_parser.rb requires a lot of memory (~30 mb)

[schneems]

Loading the address_lists_parser.rb is the cause for about 1/3 of the startup memory for CodeTriage.

TOP: 105.7266 MiB
  mail/parsers: 31.9609 MiB
    mail/parsers/address_lists_parser: 29.7266 MiB

Using the derailed_benchmarks gem on codetriage/codetriage.

It looks like it is generated by a parser generator Is there some way we can reduce that memory overhead?

[ahorek]

There's an option to recompile ragel sources for lower memory consumption, but it does hurt performance, see #1215

@jeremy any plans to support ragel 7?

[schneems]

The Ragel Bitmap PR looks pretty promising. Doesn’t look like it affects performance.

I tested with derailed and instead of 30 mb I’m seeing about 5mb. Which is a pretty significant improvement.

Edit, it totally affects performance

[jeremy]

Remember #812 → #815 → #912? 😅

Lazy-loading the parsers is desirable, considering that most usage is building and delivering messages, not parsing them. But the lazy-loading has to work consistently.

[ahorek]

Well, it depends. For our helpdesk we use the mail gem for parsing incoming messages a lot.
Ragel never used to generate a performant code in ruby, but the question is if 30MB per instance really matters these days...

[schneems]

😅 Sorry for the duplicate issue spam. I was working with someone on derailed benchmarks and mis-remembered that somehow I had gotten away without the memory increase for https://www.codetriage.com.

While it doesn't look like there is an obvious replacement (that satisfies all memory and speed requirements) it did generate some good discussion and eyeballs in #1343 whether any changes end up getting merged or not, there's some new techniques in there that i've not seen before.

the question is if 30MB per instance really matters these days...

It matters more for memory constrained implementations, FaaS and PaaS. Usually, there's a line somewhere and if you're over even a few mb, you'll still start to swap and incur enormous perf penalties.

I certainly understand that for some absolute performance is important. I also understand that one of the goals of the project is to not have a c-extension dependency.

Thanks for your time and maintenance!