Upgrade PostgreSQL from 9.6 to something more modern

[tanmaymohan]

Original Report

Problem Statement

Hello

I have been using Sentry 21.9.0 for production use for a python based api , and I am using AWS RDS Postgres 9.6 as the database server for the sentry application. I have been getting a warning message from AWS regarding the EOL date of Postgres9.6 and its deprecation is nearby but your docker-compose and other tools still list postgres:9.6 as the db server. Are there any plans for upgrade to a newer version of postgres ?

Solution Brainstorm

Here's an excerpt : https://forums.aws.amazon.com/ann.jspa?annID=8499

Update from Sentry

1. Hosting on Postgres 10, 11, 12, 13 and 14 seems to work, but should be considered experimental and second-class until we upgrade SaaS away from Postgres 9.6, which is not currently prioritized or scheduled.
2. We run sentry CI against Postgres 9.6 and 14.
3. We do not officially support Sentry on RDS, but it's an obvious and reasonable customization, so we'd rather not break it if we don't have to.

[BYK]

Heya, thanks a lot for the feedback! We are aware of this and are actively working on it: https://forum.sentry.io/t/roadmap-postgresql-update/14161/2?u=byk

[marin-h]

Hey, folks. Is there any estimated for this? We are starting to look into Hosted Sentry as an alternative, since RDS is deprecating Postgres v9 in Jan 2022.

Also, in that forum BYK linked, someone states that a custom postgres plugin is being added. This will make onpremise non-compatible with RDS and other alike services. Are planning on offering any alternative / backwards compatibility to avoid the need of using that custom plugin?

[chadwhitacre]

Hey @marin-h, thanks for escalating (here, too).

Is there any estimated for this?

No, but we're close enough now that we had better get on this. @jasonious Can you fill us in on the current state of Sentry on Postgres >9 from the Ops perspective?

We are starting to look into Hosted Sentry as an alternative

Awesome, that is certainly a solid plan. :)

RDS is deprecating Postgres v9 in Jan 2022

I believe the forum link you shared @tanmaymohan is the source of truth for this, yes?

If you do not upgrade your databases before January 18, 2022, RDS will upgrade your PostgreSQL 9.6 databases to 12 during a scheduled maintenance window between January 18, 2022 00:00:01 UTC and February 22, 2022 00:00:01 UTC. On February 22, 2022 00:00:01 AM UTC, any PostgreSQL 9.6 databases that remain will be upgraded to version 12 regardless of whether the instances are in a maintenance window or not.

That certainly sounds drastic. Question: has anyone tried running Sentry on PostgreSQL 12 already? How did it go?

[marin-h]

That certainly sounds drastic. Question: has anyone tried running Sentry on PostgreSQL 12 already? How did it go?

I read about users trying the upgrade in Sentry on-premise forum saying it didn't worked (they said something about missing columns upon creating / altering tables)

[chadwhitacre]

@marin-h Link? Not jumping out at me on https://forum.sentry.io/t/roadmap-postgresql-update/14161.

[marin-h]

Found it! It was on this thread

[chadwhitacre]

@marin-h Good find! So it looks from one report at least that there are indeed some issues running against Postgres 11 (cc: @dcramer):

I faced a lot of conflicts during the installation process such as unable to find a specific column, unable to alter relations, and etc.

That said, 11 is two hops. The minimum requirement to meet the February 22 RDS deadline is Postgres 10. Does Sentry work against Postgres 10? 🧐

Sidebar on CDC

There was a concern (1, 2) that the introduction of CDC will mean Sentry no longer works on RDS because CDC introduces a custom extension, but afaict the only extension required by CDC is the well-established wal2json, which RDS does install for Postgres 10.x. CDC itself is a Python service that consumes the replication log produced by wal2json and posts messages to Kafka. We started installing wal2json as of 21.6.0. We haven't rolled out CDC yet, but when we do it will simply be another entry in docker-compose.yml.

@fpacifici Please correct me if I'm wrong.

Soooooo ... Postgres 10? 😬

Postgres 9.6 is EOL as of last week. Last word we have on upgrading from the Sentry side is from June:

Upgrading PostgreSQL is definitely on our To Do list, but there are a few bumps in the road. The first is that a glibc upgrade causes big problems for sort in PostgreSQL, which we have to work around first. We have changed our replication very recently as a first step towards getting to a newer PostgreSQL. As for exactly when that will happen, I’m not totally certain, but it is actively being worked on.

As you can imagine, we have to be very careful with this as it’s fine and dandy for us to manage our own PostgreSQL, but to introduce a breaking change into On-Prem Sentry would be bad for everyone. Once we get it sorted for our own installations, we’ll get On-Prem Sentry sorted as well.

@jasonious @dbennett-sentry Can you bring us up to speed on our internal timeline for a Postgres upgrade? If we can't commit to February 22, then the community will need to step up with fixes to meet the RDS deadline.

[marin-h]

That said, 11 is two hops. The minimum requirement to meet the February 22 RDS deadline is Postgres 10. Does Sentry work against Postgres 10? monocle_face

Haven't tried it yet, did you? I might give it a shot next week

Sidebar on CDC

There was a concern (1, 2) that the introduction of CDC will mean Sentry no longer works on RDS because CDC introduces a custom extension, but afaict the only extension required by CDC is the well-established wal2json, which RDS does install for Postgres 10.x. CDC itself is a Python service that consumes the replication log produced by wal2json and posts messages to Kafka. We started installing wal2json as of 21.6.0. We haven't rolled out CDC yet, but when we do it will simply be another entry in docker-compose.yml.

@fpacifici Please correct me if I'm wrong.

This is great news!

[chadwhitacre]

Haven't tried it yet, did you? I might give it a shot next week

I have not, no. If you're able to try it out that would be great! :-)

[pharindoko]

Hmm I do have the same issue @marin-h
Will try to test the upgrade to Postgre 10 in December / January as well.

@BYK @chadwhitacre
Can you help to test this or check internally with the team if anyone tried it out ?

[chadwhitacre]

Will try to test the upgrade to Postgre 10 in December / January as well.

Awesome, keep us posted. 👍

Can you help to test this or check internally with the team if anyone tried it out ?

I pinged @jasonious internally to see if we can get an update here. Also fyi @BYK has moved on to his next adventure (as they say ;), he's still hanging around here but no longer full-time with Sentry.

[chadwhitacre]

I was able to connect with @jasonious, we don't really have any movement on our side on this or plans to tackle this in the near term. If folks need this to meet the AWS deadline we'll have to see the community step up.

[lindsaygrace]

I've been following this for a while, and this is quite worrisome. We're currently using RDS however like many we go into a forced break over the holidays, which results in even less time to implement any significant architectural changes, which it's sounding like will be required.
It's quite disappointing to see communication that we'll need to solve this ourselves with such short notice. There should be some rather public communication with the community about this asap.

[chadwhitacre]

@lindsaygrace et al. I've pinned this issue in the repo for visibility, and I've started #1213 to start to get a sense of how much work this will be. Can someone else try out #1213 and see if it works for you?

[chadwhitacre]

#1213 ran green. 👍

Testing out with sentry CI in getsentry/sentry#30782 ... 🤞

[Apollon77]

When I remember correctly there was an unintentionally upgrade try som time ago and the "db migration in install.sh" was the "topic" why it was rolled back or such ... but this is somehow deep from rememberings

[chadwhitacre]

@Apollon77 Was this on an old forum thread or something? Or your personal experience? Or ... ?

[Apollon77]

This is exactly the question ... I now searched for several time but can not find it anymore :-( Sorry for pot. confusion. maybe it was just a wrong remembering? :-(

In fact the DB migration will in fact be the interesting point in that I assume

[chadwhitacre]

Checks passed on getsentry/sentry#30782! 💃

Here's what I propose:

1. Let's add Postgres 10 to the test matrix for sentry CI, so that we test against both Postgres 9 and 10.
2. When that is in place, let's merge Upgrade to Postgres 10 #1213, bumping the default version in self-hosted to Postgres 10.

It seems that we should be able to get that done for the 22.1.0 release in January. No hard promises on date. 😬 With green CI those who are on RDS should consider upgrading now without blocking any further.

What do we need to consider in terms of upgrade instructions?

Also, I would love to see someone upgrade to Postgres 10 asap so we can get some real-world signal. Who is going to be our pioneer / guinea pig / beta tester? 😁

[Apollon77]

What do we need to consider in terms of upgrade instructions?

In my eyes the "install.sh" needs to handle that automatically - or at least need to "block" an update and refer to instructions (I'm no PostgreSQL expert, so I can not tell what needed)

[chadwhitacre]

There should be some rather public communication with the community about this asap.

We can put this in the changelog for 22.1.0 one way or another, even if we don't land any code changes in that version we can say "we tested this on Postgres 10 and it seems to work, if you're using RDS be aware of the Jan 18 soft deadline and the Feb 22 hard deadline to upgrade."

Between that and this ticket what further communication should we do?

[fpacifici]

@chadwhitacre Sorry for missing that detail before.
What sentry install is not the standard wal2json version but a fork of it. https://github.com/getsentry/wal2json. The version in master would not work without updates on Sentry side.

[chadwhitacre]

Thanks @fpacifici, scheduling a discussion internally to coordinate. Will report back here ...

[chadwhitacre]

Via @aminvakil at #1213 (comment), consolidating here for visibility:

I'm not sure if sticking to a specific minor version of postgres has more upsides than downsides.

https://www.postgresql.org/support/versioning/:

We always recommend that all users run the latest available minor release for whatever major version is in use.

Major versions usually change the internal format of system tables and data files. These changes are often complex, so we do not maintain backward compatibility of all stored data. A dump/reload of the database or use of the pg_upgrade module is required for major upgrades. We also recommend reading the upgrading section of the major version you are planning to upgrade to. You can upgrade from one major version to another without upgrading to intervening versions, but we recommend reading the release notes of all intervening major versions prior to doing so.

Upgrading to a minor release does not normally require a dump and restore; you can stop the database server, install the updated binaries, and restart the server. For some releases, manual changes may be required to complete the upgrade, so always read the release notes before upgrading.

While upgrading will always contain some level of risk, PostgreSQL minor releases fix only frequently-encountered bugs, security issues, and data corruption problems to reduce the risk associated with upgrading. For minor releases, the community considers not upgrading to be riskier than upgrading.

[chadwhitacre]

Circling back from our internal call:

1. We do not officially support Sentry on RDS, but it's an obvious and reasonable customization, so we'd rather not break it if we don't have to.
2. The wal2json plugin that we use is indeed a fork, but the system that depends on it is not used in production yet, and is unlikely to be used for "quite some time." Sentry should function fine without it for the foreseeable future.
3. Once we come to depend on wal2json for real, our intention is to support both the original version and our fork of it (which contains some optimizations that are not relevant except at extremely large scale, and may not be relevant even then).
4. We intend to add Postgres 10 to our CI matrix for sentry, and to block new PRs on any regressions against it.
5. Hosting on Postgres 10 should be considered experimental and second-class until we migrate SaaS to Postgres 10, which is not currently prioritized or scheduled.

We always recommend that all users run the latest available minor release for whatever major version is in use.

For our use-case where we have a battle-tested SaaS installation, we should pin to the Postgres version in that environment as our default, though others are of course free to target different versions for their own circumstances.

"install.sh" needs to handle that automatically - or at least need to "block" an update and refer to instructions

It will be the latter.

bumping the default version in self-hosted to Postgres 10.

I'll backpedal now and say that Postgres 9.6 should remain the default until SaaS is on 10. Once we flip the self-hosted default to 10 for new instances, it will still require a manual dump and reload to upgrade existing instances just as it does now. We can write docs for this starting now.

That's the status for now. I added an update to the ticket description here. If you have suggestions for other places to communicate about this, lmk.

It would be super-helpful for someone from the community to take the plunge and migrate to Postgres 10 and report back with their experience. 🙏

Thanks everyone! :)

[marin-h]

Hey, I've just accomplished postgres v10.17 upgrade (from v9.6.22) following RDS official guide and everything seems to be working smoothly. I'm using getsentry/sentry:20.11.1 image.

I'll be reporting here if anything comes up.

[chadwhitacre]

Woo-hoo! Thanks for taking this on and for sharing back out @marin-h! That's good news! 👍

[chadwhitacre]

I believe the last action item to complete before closing this ticket is to add Postgres 10 to the CI matrix for sentry. Upgrading SaaS is so far in the future that we should open a fresh ticket here when the time comes.

[chadwhitacre]

If/when we get to upgrading to Postgres 10, we should pay attention to the install/upgrade-postgres.sh script. See also: #1213 (review) ff.

[aminvakil]

FWIW https://github.com/tianon/docker-postgres-upgrade/tree/master/9.6-to-10 is available.

[chadwhitacre]

We can put this in the changelog for 22.1.0 one way or another

Done.

[chadwhitacre]

We previously communicated in the PostgreSQL 9.6 End-of-Life forum post [1] that we would begin automatic upgrades on January 18, 2022. We have decided to extend the end-of-life process for all customers.

The new date is April 26.

https://forums.aws.amazon.com/ann.jspa?annID=9092

[pharindoko]

Hey,

I already changed my prod and dev stage to the latest sentry version and afterwards from aws rds postgre 9.6 to 10.19 a week ago.

I haven't detected any issues yet.

[chadwhitacre]

Nice! Thanks for posting back, @pharindoko. I've added a shout-out along with @marin-h in our monthly newsletter which should be going out in the next few days. :)

[chadwhitacre]

Lolsob I just found this ticket over in the sentry repo: getsentry/sentry#12673.

There we have reports of Sentry running on Postgres 11 and 12. I've updated the Update in the description here accordingly.

I believe the last action item to complete before closing this ticket is to add Postgres 10 to the CI matrix for sentry.

I guess let's look at 11 and 12 as well when we get to this.

[pavelbrylov]

For what it's worth, we've configured Sentry to run against AWS RDS Postgres 13.3 and it seems to work fine? Or should I test specific scenario that doesn't work on newer versions?

[chadwhitacre]

Thanks for the report, @pavelbrylov! I haven't heard of specific failures we would want to test. I've updated the ticket description to note your success with 13. 👍

[chadwhitacre]

We started running CI against 14 a few weeks ago, in getsentry/sentry#33041.

[chadwhitacre]

I believe the last action item to complete before closing this ticket is to add Postgres 10 to the CI matrix for sentry.

Gonna go ahead and close this out now that we have getsentry/sentry#33041. Looks like we're on our way! 💃

[chadwhitacre]

Picking up with #1610 now that SaaS is on 14.