Breaking changes in PostgreSQL 12's libpq #147
Comments
Since libpq 12 includes a breaking change (more strict validation of connection parameters) which causes connection errors at runtime for customers with invalid database configuration settings. This unblocks new stack image releases until an assessment of the number of customers affected/outreach can be performed. The version is being pinned using the approach here: https://wiki.postgresql.org/wiki/Apt/FAQ#I_want_libpq5_for_version_X.2C_but_there_is_only_version_Y_in_the_repository Cedar-14's build script did not need to be updated, since PostgreSQL 12 is not available in the `trusty-pgdg` archive (due to Ubuntu 14.04 EOL). Refs #147 / W-6780695.
Since libpq 12 includes a breaking change (more strict validation of connection parameters) which causes connection errors at runtime for customers with invalid database configuration settings. This unblocks new stack image releases until an assessment of the number of customers affected/outreach can be performed. The version is being pinned using the approach here: https://wiki.postgresql.org/wiki/Apt/FAQ#I_want_libpq5_for_version_X.2C_but_there_is_only_version_Y_in_the_repository Cedar-14's build script did not need to be updated, since PostgreSQL 12 is not available in the `trusty-pgdg` archive (due to Ubuntu 14.04 EOL). Refs #147 / W-6780695.
|
The Leaving this issue open to track resolving the PG12 compatibility issues at some point in the future. |
|
@edmorley so these are clearly integer values, what do customers put there?:) |
|
There were two variations seen (both for Rails apps, using connect_timeout: <%= ENV['SOME_VAR'] || 5 %># some comment...the lack of whitespace before the comment means And for the other: connect_timeout: 15sWhat made this hard for the customer to debug was the Ruby pg gem swallowing the real error. I've filed ged/ruby-pg/issues/302 for this, but even if it's fixed it's going to be some time before customers update to the new version. |
|
So there's some good news about the confusing Ruby pg gem error message (that makes debugging harder) - it turns out it's a bug in libpq, for which the author of the Ruby pg gem has kindly created a patch. Most importantly this means we can pick up the fix ourselves (once released), rather than needing to encourage/wait for customers to update their app's pg gem. Whilst this doesn't make the libpq 12 changes any less breaking (for customers with invalid configs), it at least gives customers an actionable reason that they can self-serve without having to wait for support ticket assistance. |
|
The libpq fix that results in an improved error message in ruby-pg (and likely others) landed in: There hasn't yet been a new libpq release since then, from what I can tell. |
|
@edmorley Thanks for the update here. We're walking through other breaking changes in 12.0 and we'll continue to track this. It will likely will be in the next patch release but we'll stay tuned as well |
|
@edmorley can we have a separate stack images for pg12 specifically until we clean this up, it can take a bit and we would like customers to start using them? |
|
@edmorley we started something here - https://salesforce.quip.com/azvDAzWyI3NO |
|
@lyubo1 Hi! Just checking you'd seen @schneems and @genslein's notes here? |
|
@edmorley thanks, have not! |
This undoes the pinning of libpq added in #148 and additionally updates `postgresql-client` and `postgresql-server-dev` to the PG12 releases, to support Heroku Postgres 12. Since #148 the upstream Postgres project has improved the libpq 12 error handling, such that it now provides a clearer error message when the validation of connection parameters fail, which makes it easier for customers to identify that it's due to their app having an invalid configuration. The build scripts for Cedar-14 have not been modified since the upstream Postgres project has not released PG12 versions of the packages for Ubuntu-14 given it is EOL. Release notes: https://www.postgresql.org/docs/12/release-12.html Refs W-7113082. Closes #147.
The stack images currently install the PostrgreSQL client, library and headers from the upstream PostgreSQL APT repository (
apt.postgresql.org) rather than the less frequently updated packages in Ubuntu's APT repository.This means that as new major Postgres versions are released, the next stack image release will include updated major versions of
libpq5andlibpq-dev(the clients themselves use versioned package names such aspostgresql-client-11so have to be explicitly updated).New major versions of libpq are generally backwards compatible, however libpq v12 includes the following breaking change that makes the validation of connection parameters more strict:
postgres/postgres@e7a2217
Unfortunately this breaking change wasn't listed in the PostgreSQL 12 release notes, so we weren't able to factor it into the stack release review process.
This meant the latest stack image update caused database connection errors at runtime for a couple of customers, since their Rails apps were using invalid values for connection parameters (such as
connect_timeouthaving trailing non-numeric characters). (Previously with libpq v11 these invalid values were being silently ignored.)As such we:
PG::UnableToSend: no connection to the serverDetermining how to best roll out the PG 12 client libraries will be more involved, so in the short term we should pin to v11 so that stack image releases are unblocked. After that someone will need to investigate further as to how widespread invalid app configurations are and work on a rollout plan.
CC @heroku/dod-infra since they will need to be aware of this if wanting to upgrade to PG12 in the stack image in the future. For more info see the investigation notes here.
The text was updated successfully, but these errors were encountered: