undefined symbol: _ZTVN10__cxxabiv117__class_type_infoE

[pat]

I'm trying to install the mysql2 gem on Heroku's latest stack heroku-20 (Ubuntu 20.04), but have it compile against MySQL 5.7. I've added the libmysqlclient-dev_5.7.32 package via the apt buildpack, but I'm not convinced it's being compiled against.

Whenever the gem is loaded via require "mysql2", I get the error:

/app/vendor/bundle/ruby/2.7.0/gems/mysql2-0.5.3/lib/mysql2/mysql2.so: undefined symbol: _ZTVN10__cxxabiv117__class_type_infoE - /app/vendor/bundle/ruby/2.7.0/gems/mysql2-0.5.3/lib/mysql2/mysql2.so

Heroku's stack does have the MySQL 8 library as well, in /lib/x86_64-linux-gnu/. The apt-provided (5.7) library is available in /app/.apt/usr/lib/x86_64-linux-gnu, and that path is first in $LD_LIBRARY_PATH. It's also what mysql_config points to.

$ mysql_config
Usage: /app/.apt/usr/bin/mysql_config [OPTIONS]
Compiler: GNU 7.5.0
Options:
        --cflags         [-I/app/.apt/usr/include/mysql ]
        --cxxflags       [-I/app/.apt/usr/include/mysql ]
        --include        [-I/app/.apt/usr/include/mysql]
        --libs           [-L/app/.apt/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lm -lrt -latomic -ldl]
        --libs_r         [-L/app/.apt/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lm -lrt -latomic -ldl]
        --plugindir      [/usr/lib/mysql/plugin]
        --socket         [/var/run/mysqld/mysqld.sock]
        --port           [0]
        --version        [5.7.32]
        --libmysqld-libs [-L/app/.apt/usr/lib/x86_64-linux-gnu -lmysqld -lpthread -lm -lrt -latomic -ldl -lcrypt -laio -lnuma]
        --variable=VAR   VAR is one of:
                pkgincludedir [/app/.apt/usr/include/mysql]
                pkglibdir     [/app/.apt/usr/lib/x86_64-linux-gnu]
                plugindir     [/usr/lib/mysql/plugin]

The output from ldd doesn't mention MySQL at all though:

$ ldd /app/vendor/bundle/ruby/2.7.0/gems/mysql2-0.5.3/lib/mysql2/mysql2.so
	linux-vdso.so.1 (0x00007ffe07b31000)
	libruby.so.2.7 => not found
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3f42c90000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3f42c8a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3f42a98000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3f42a7d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f3f43373000)

The other complication is that Heroku's build process occurs in a separate randomly generated path (so that's where the gem and the apt package first exist) - which means the gem's .so file lives temporarily at something like /tmp/build_bd61275a/vendor/bundle/ruby/2.6.0/gems/mysql2-0.5.3/lib/mysql2/mysql2.so. This means I can't hardcode library paths as part of the mysql2 gem compilation, because the paths change. It also means I can't auto-require the mysql2 gem as part of a Rails assets compilation, because I hit the same error, but with this build path instead of in /app/. To get around this, it's listed as gem "mysql2", require: false in my Gemfile.

I realise this might be a bit of an edge case, but if anyone can provide pointers to help debug, that'd be great. I'm a bit out of my depth here with compiled dependencies and library paths and such. Has anyone seen the error message before? Does it mean that the gem's compiled against the wrong version of MySQL? Or something else? Is there any way to confirm which MySQL library files it's using?

(And compiling for MySQL 8+ isn't an option - I need mysql2 to talk to Sphinx v2.2.11, which predates MySQL v8).

[pat]

I've spent a bit more time on this today, and it seems the issue - at least, the visible issue - is that it wasn't picking up libstdc++, even though it is present. The fix, at least on Heroku, is to set the LD_PRELOAD environment variable (which in my cases wasn't previously set to anything). I also need to ensure mysql2 picks up the apt-buildpack-provided mysql_config. So:

$ heroku config:set LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 BUNDLE_BUILD__MYSQL2=--with-mysql-config

Along with https://cdn.mysql.com/Downloads/MySQL-5.7/libmysqlclient-dev_5.7.32-1ubuntu18.04_amd64.deb in my Aptfile, I'm finding that this is enough to get mysql2 working on Heroku's heroku-18 buildpack. Something's not quite right for heroku-20 though, but that's a different problem. The focus of this issue was the undefined symbol error, and setting LD_PRELOAD fixed that.