"/vendor/bundle/gems/pg-1.2.2/lib/pg_ext.so: undefined symbol: PQresultMemorySize" error occurs on CircleCI / Heroku Private Spaces

[spemmons]

This has just started happening on CircleCI in the last few days with all else being the same on our Rails 6.0.2.1 application. We've tried using Postgres 11 an 10. On CCI, we're running using their alpine-ram pre-created docker images for PG.

In searching for clues, we ran across the following in Heroku's support incidents: https://status.heroku.com/incidents/1940

In the meantime, we are going to try the recommendation to back down to 1.1.4

[larskanis]

PQresultMemorySize is new in PostgreSQL-12, so you obviously built Ruby-pg with version 12 but try to use it on version 11. This expectingly results in a runtime linker error.

[spemmons]

we have been using pg 1.2.2 since we upgraded to Rails 6.0.2.1 a few weeks ago... we had not seen any issues either locally, on our production ubuntu servers, or on CCI until this error started occurring on CCI a few days ago ... we did not build with version 12 in any environment, so not sure what triggered this ... i'm not sure how CCI builds their "alpine-ram" docker images but they have them for each version of PG as well ... it could be that the CCI ruby-2.6.5-alpine docker image was (inadvertently?) built with the PG 12 client? ... not sure

in any case, backing down to 1.1.4 solved our problem

it would be nice if 1.2.2+ could be backward compatible to earlier versions of PG

[larskanis]

in any case, backing down to 1.1.4 solved our problem

Alternatively you could upgrade your runtime environment to PostgreSQL-12. But in any case this build/runtime composition is broken.

it would be nice if 1.2.2+ could be backward compatible to earlier versions of PG

It's not pg's fault. pg-1.2.x is compatible down to PostgreSQL-9.2. But mixing different major PostgreSQL versions between build and runtime isn't a supported use case.

[spemmons]

unfortunately, i'm at a big company and OPS is responsible for PG version, not our team, so we're stuck w/ PG 10/11 as options

i'm not clear on where the mixing of versions is taking place ... i've raised a support ticket w/ CCI, but not sure if/when they will do anything about it

from my link about heroku, apparently they have a "private spaces" thing that has the same problem hence their recommendation to use 1.1.4

sorry for my ignorance, where do you think the "mixing" of versions is taking place here? ... maybe you need some additional pieces of info about our rails config? ... just let me know

[cbandy]

@spemmons the pg gem has native extensions which utilize the libpq C library provided by PostgreSQL.

With any program involving C, there is a build step before anything can run. The pg gem is built when it is installed (or updated) by bundle install or gem install. The machine that builds the gem must first have libpq installed so that the gem can be "linked" to it and use its functionality.

Once both are installed, linked, etc, a Ruby program can run utilizing the gem which in turn utilizes the library.

Unfortunately, building (installing, linking) and running can happen on different machines. In this case, the machine that built the gem had libpq for PostgreSQL 12 installed while the machine that runs your Ruby program has libpq for PostgreSQL 11 installed. This is the mixing @larskanis is describing.

If you are able to run commands in your CI environment as part of your build, it could be that gem pristine pg rebuilds the gem with the version of libpq that is available there. (@larskanis please correct me if I'm wrong.) It could also be that the environment lacks the tools to rebuild the gem.

Something else to note is that libpq for PostgreSQL 12 is able to talk to a PostgreSQL 9, 10, or 11 server. You do not need to match the client and server versions in this case.

[spemmons]

curious... we are doing the bundle install on the CircleCI server which should be installing the underlying C code on a docker image that purports to have a the specific version of postgres installed ... so it's strange that postgres 12 dependencies would exist ... that said, CircleCI also has a Ruby docker image that we also use ... it appears that probably THAT image MAY have the libq for postgres 12 installed ... that's something i can bring to their attention (if they ever respond :) ) .. thanks for your help

[romanbsd]

I found the following workaround. It hardcodes the path to the shared library in the pg_ext.bundle:

install_name_tool -change libpq.5.dylib /Library/PostgreSQL/12/lib/libpq.5.12.dylib pg_ext.bundle

[larskanis]

This issue seems to be solved. From the ruby-pg view we need equal prerequisites at gem install and gem usage time. Otherwise any library checks done at gem install would be useless.

[lorint]

For other folks who may tread these murky waters ...
Try bundle pristine pg to recompile the native code used by the pg gem.