Added support for Debian 10 (buster)

[ndbroadbent]

Fixes #16

I just needed to rename libgdbm4 to libgdbm6, then everything was compiled and packaged without any problems.

I also updated the README to include the debian release names: 9 (stretch), 10 (buster)

[ndbroadbent]

Ahh, I'm just installing gems and compiling all of the native extensions, and I got this crash with the sq_mini_racer gem (a mini_racer fork for the Sqreen ruby agent).

This should be nothing to worry about, because I can install the latest mini_racer with no problems.

Step 34/37 : RUN gem install sq_mini_racer -v 0.2.5.0.1.beta2
 ---> Running in f4b8537e912f
Building native extensions. This could take a while...
ERROR:  Error installing sq_mini_racer:
	ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension
/usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/bin/ruby -I /usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/lib/ruby/site_ruby/2.5.0 -r ./siteconf20191117-7-oq5dj2.rb extconf.rb
detected platform x86_64-linux => x86_64-linux
looking for libv8-7.3.492.27.1-x86_64-linux.gemspec in installed gems
libv8-7.3.492.27.1-x86_64-linux.gemspec not found in installed gems
looking for libv8-7.3.492.27.1-x86_64-linux in /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension/vendor
libv8-7.3.492.27.1-x86_64-linux not found in /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension/vendor
downloading libv8-7.3.492.27.1-x86_64-linux.gem to /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension/vendor
looking for libv8-7.3.492.27.1-x86_64-linux in /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension/vendor
looking for libv8-7.3.492.27.1-x86_64-linux/lib/libv8.rb in /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension/vendor
checking for -lpthread... yes
creating Makefile

current directory: /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension
make "DESTDIR=" clean

current directory: /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2/ext/mini_racer_extension
make "DESTDIR="
compiling mini_racer_extension.cc
cc1plus: warning: command line option ‘-Wimplicit-int’ is valid for C/ObjC but not for C++
In file included from /usr/lib/gcc/x86_64-linux-gnu/8/include/xmmintrin.h:34,
                 from /usr/lib/gcc/x86_64-linux-gnu/8/include/x86intrin.h:33,
                 from simdutf8check.h:8,
                 from mini_racer_extension.cc:37:
/usr/lib/gcc/x86_64-linux-gnu/8/include/mm_malloc.h:34:16: error: declaration of ‘int posix_memalign(void**, size_t, size_t) throw ()’ has a different exception specifier
 extern "C" int posix_memalign (void **, size_t, size_t) throw ();
                ^~~~~~~~~~~~~~
In file included from /usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/ruby/missing.h:25,
                 from /usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/ruby/defines.h:154,
                 from /usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/ruby/ruby.h:29,
                 from /usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/ruby.h:33,
                 from mini_racer_extension.cc:22:
/usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/jemalloc/jemalloc.h:42:29: note: from previous declaration ‘int posix_memalign(void**, size_t, size_t)’
 #  define je_posix_memalign posix_memalign
                             ^~~~~~~~~~~~~~
/usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/jemalloc/jemalloc.h:140:21: note: in expansion of macro ‘je_posix_memalign’
 JEMALLOC_EXPORT int je_posix_memalign(void **memptr, size_t alignment,
                     ^~~~~~~~~~~~~~~~~
cc1plus: warning: unrecognized command line option ‘-Wno-self-assign’
cc1plus: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
cc1plus: warning: unrecognized command line option ‘-Wno-parentheses-equality’
make: *** [Makefile:211: mini_racer_extension.o] Error 1

make failed, exit code 2

Gem files will remain installed in /usr/local/bundle/gems/sq_mini_racer-0.2.5.0.1.beta2 for inspection.
Results logged to /usr/local/bundle/extensions/x86_64-linux/2.5.0/sq_mini_racer-0.2.5.0.1.beta2/gem_make.out

This native extension was compiling fine using the same fullstaq ruby version for Debian 9.

UPDATE: I just saw this changelog entry for jemalloc 5.2.1:

Fix size 0 handling in posix_memalign(). This regression was first released in 5.2.0. (@interwq)

I will try to compile with the latest jemalloc and see if that fixes it.

UPDATE 2: Great! Updating to jemalloc 5.2.1 allows me to compile the native extension without any errors:

$ gem install sq_mini_racer -v 0.2.5.0.1.beta2
Building native extensions. This could take a while...
Successfully installed sq_mini_racer-0.2.5.0.1.beta2
1 gem installed

I also realized that this might not be an isolated incident specifically for sq_mini_racer, and other gems might fail with the same error. Another thing I noticed is that Debian 9 uses GCC 6, and Debian 10 has been updated to use GCC 8.

I'm also not sure if using the very latest version is a bit dangerous. So maybe there's a safer version of jemalloc somewhere between 3.6.0 and 5.2.1.

Anyway, I will be testing this out in test/staging/production over the coming weeks, and will update if I run into any more issues.

[ndbroadbent]

Hmm, I'm trying to build Ruby 2.6.5 now, so I ran: ./build-all-packages build:ruby-2.6.5:debian-10:jemalloc

But it looks jemalloc is missing from the package:

$ ruby --version
ruby 2.6.5p114 (2019-10-01 revision 67812)+jemalloc [x86_64-linux]

$ ruby -r rbconfig -e "RbConfig::CONFIG['LIBS'].include?('jemalloc') \
  ? puts('Ruby is compiled with jemalloc.') \
  : raise('jemalloc is missing from Ruby!')"
Traceback (most recent call last):
-e:1:in `<main>': jemalloc is missing from Ruby! (RuntimeError)

$ ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS'].inspect"
"-lm "

Not sure why that's happening.

[ndbroadbent]

@FooBarWidget Would it be possible to add a "sanity check" after compiling/packaging Ruby, to make sure that this can be caught a bit earlier? (I'm just not too familiar with these scripts, so I don't know the best place to put that.)

EDIT: I was also wondering about the rationale behind jemalloc 3.6.0, instead of the latest 5.x version. I was searching on Google for things like "stable jemalloc for Ruby", but I couldn't find any resources. Has 3.6.0 gone through more testing, and are they still backporting fixes/security patches for 3.x?

[ndbroadbent]

Ah I finally figured it out, from the make install output:

builder@aeb35c9e8d93:~/ruby-src/ruby-2.6.5$ run make install DESTDIR="$DESTDIR"
+ make install DESTDIR=/home/builder/ruby-inst
generating x86_64-linux-fake.rb
x86_64-linux-fake.rb updated
	BASERUBY = echo executable host ruby is required.  use --with-baseruby option.; false
	CC = gcc
	LD = ld
	LDSHARED = gcc -shared
	CFLAGS = -O3 -ggdb3 -Wall -Wextra -Wdeclaration-after-statement -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wrestrict -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -std=gnu99  -fPIC
	XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -DCANONICALIZATION_FOR_MATHN
	CPPFLAGS =   -I. -I.ext/include/x86_64-linux -I./include -I. -I./enc/unicode/12.1.0
	DLDFLAGS = -Wl,--compress-debug-sections=zlib -Wl,-soname,libruby.so.2.6  -fstack-protector-strong
	SOLIBS = -lz -lpthread -lrt -lrt -ljemalloc -ldl -lcrypt -lm
	LANG =
	LC_ALL =
	LC_CTYPE =

I just needed to rename RbConfig::CONFIG['LIBS'] to RbConfig::CONFIG['SOLIBS'].

$ $RUBYPATH/ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']"
-lm

$ $RUBYPATH/ruby -r rbconfig -e "puts RbConfig::CONFIG['SOLIBS']"
-lz -lpthread -lrt -lrt -ljemalloc -ldl -lcrypt -lm

LIBS previously included -ljemalloc in Ruby 2.5.7, but it looks like this has moved to SOLIBS for Ruby 2.6.x.

I will update the test to check for both 'LIBS' and 'SOLIBS':

$ $RUBYPATH/ruby -r rbconfig -e "puts ['LIBS', 'SOLIBS'].map{ |k| RbConfig::CONFIG[k] }.join"
-lm -lz -lpthread -lrt -lrt -ljemalloc -ldl -lcrypt -lm

[FooBarWidget]

I'm also not sure if using the very latest version is a bit dangerous. So maybe there's a safer version of jemalloc somewhere between 3.6.0 and 5.2.1.

That's a bummer. Newer Jemalloc versions are problematic. Jemalloc > 3 don't result in as much memory usage reduction as v3. The cause is still unknown.

[ndbroadbent]

Ahhh I see! I just saw this tweet from Nate Berkopec:

FYI: Jemalloc 5.x+ should work well with the newly-released Ruby 2.6.0, due to Ruby disabling Transparent Huge Pages (https://bugs.ruby-lang.org/issues/14705). THP caused a very bad interaction with the jemalloc 4.x and, sometimes 5.x versions. Anyone running Jemalloc 5.x and Ruby 2.6?

Maybe it would be possible to use Jemalloc 5.x for Ruby >= 2.6?

[FooBarWidget]

That's indeed what he said in January, but I tested Jemalloc 5 somewhere in June and I saw that even the latest v5 uses more memory than v3, so it's not THP that's the problem.

[ndbroadbent]

Sorry I should have read more before posting that comment! I also just found your article "The status of Ruby memory trimming & how you can help with testing".

Do you have any results / updates for malloc_trim vs jemalloc?

I switched to DataDog recently for APM and metrics, and to be honest I don't really know how to use it yet! But I think I've found the average memory utilization for my containers:

This is for Ruby 2.5.7 with jemalloc 5.2.1. I'm about to deploy Ruby 2.6.5 with jemalloc 5.2.1, so I will post an update after the deploy.

[FooBarWidget]

All right, looking forward to your results.

[ndbroadbent]

I got the memory utilization result from CloudWatch in AWS:

My web containers have a memory limit of 1400 MB.

Ruby	jemalloc	Memory Utilization	Memory Usage
2.5.7	5.2.1	72.2%	1011 MB
2.6.5	5.2.1	56.7%	794 MB (-217MB / -21%)

[FooBarWidget]

So it did result in a reduction in memory usage. That's good!

It would be ideal if we can make jemalloc 3 work on Debian 10. I can give fixing it a quick try. Would you be willing to test it out to see whether it uses less memory than Jemalloc 5?

[ndbroadbent]

Oh yeah, jemalloc 3 did compile fine and Ruby was mostly fine, but I only ran into a problem when I compiled the native extension for the sq_mini_racer gem (a fork of mini_racer from the Sqreen.com service). So a lot of Ruby/Rails apps probably wouldn't come across this crash. But the source of the crash actually appeared to come from a conflict between mm_malloc.h:34:16 in gcc 8, and jemalloc.h:42:29 in jemalloc 3.

I think it might actually be the gcc 8 update that caused this, and compiling the sq_mini_racer native extension just happened to trigger it:

/usr/lib/gcc/x86_64-linux-gnu/8/include/mm_malloc.h:34:16: error: declaration of ‘int posix_memalign(void**, size_t, size_t) throw ()’ has a different exception specifier
 extern "C" int posix_memalign (void **, size_t, size_t) throw ();
                ^~~~~~~~~~~~~~

/usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/jemalloc/jemalloc.h:42:29: note: from previous declaration ‘int posix_memalign(void**, size_t, size_t)’
 #  define je_posix_memalign posix_memalign
                             ^~~~~~~~~~~~~~
/usr/lib/fullstaq-ruby/versions/2.5.7-jemalloc/include/ruby-2.5.0/jemalloc/jemalloc.h:140:21: note: in expansion of macro ‘je_posix_memalign’
 JEMALLOC_EXPORT int je_posix_memalign(void **memptr, size_t alignment,
                     ^~~~~~~~~~~~~~~~~

One solution might be for fullstaq-ruby to maintain it's own fork of jemalloc 3, and backport the new posix_memalign definition.

[FooBarWidget]

Ah okay. Let's split up the Jemalloc discussion then. Your PR looks good so I'll accept it, thanks again!