Upgrade from 23.12.1 to 24.4.1 failed

[enricolucakindle]

Self-Hosted Version

23.12.1

CPU Architecture

x86/64

Docker Version

25.0.3

Docker Compose Version

2.24.5

Steps to Reproduce

1. Shut down all services using docker compose down
2. git fetch
3. git checkout 24.3.0
4. sudo ./install.sh
5. docker compose up -d

Expected Result

Updated sentry instance running

Actual Result

Event ID

No response

[aldy505]

By any chance are you running the sudo ./install.sh and docker compose up -d on a separate shell/terminal session? That's the usual case.

If you're not running a rootless Docker, meaning you originally needed sudo to run docker, you should do sudo docker compose up -d instead.

The $BASE_IMAGE environment variable should be acquired from previous ./install.sh script, that is the usual suspect for this problem.

[enricolucakindle]

i just double checked and tried again from snapshot. i am logged in as root and running both on the same shell.

i used sudo ./install.sh and sudo docker compose up -d

here's the sentry install logfile, may it is useful:

▶ Parsing command line ...

▶ Detecting Docker platform
Detected Docker platform is linux/amd64

▶ Initializing Docker Compose ...

▶ Setting up error handling ...
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 292B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/debian:bookworm-slim
#2 DONE 0.4s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/2] FROM docker.io/library/debian:bookworm-slim@sha256:3d5df92588469a4c503adbead0e4129ef3f88e223954011c2169073897547cac
#4 DONE 0.0s

#5 [2/2] RUN set -x   && apt-get update   && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends jq   && apt-get>
#5 CACHED

Many thanks

[inoa-jboliveira]

Hi, we are having exactly the same issue, running from the same terminal both commands.
We are upgrading from 23.11 to 24.4.1

[azaslavsky]

Have you tried sudo as well with no luck?

[inoa-jboliveira]

@azaslavsky Yes, I ran everything as root (sudo su)

[azaslavsky]

Are you willing to try docker system prune -a on this device, to clear any residual docker state? Note that this shouldn't cause loss of data in the volumes themselves, but state in the running containers will get dropped.

How long have you had this install for?

[inoa-jboliveira]

Hi @azaslavsky, I will give it a try after hours since this is a production server. This install we had for a couple years now. I believe we upgraded sentry 3 or 4 times on it.

Since the install removes all containers, wouldn't it somewhat reset or prune things same as this command you suggest?

[enricolucakindle]

we used this installation since december 2023 and updated once since then.

I will try that too this or next week with a clone of productive server.

[hubertdeng123]

Did the install script completely succeed? Maybe values in your .env file are not being picked up properly to your docker compose file? Specifically for VROOM?

[inoa-jboliveira]

@hubertdeng123 No, it did not work for me. FYI I'm running ubuntu 20.4 LTS with everything up to date. I did try the docker prune command.

I believe the ./install.sh script is simply dying too soon. In other versions it would show the docker compose command at the end, but now it doesn't show anything after this warning about memcache. Below is the end of the command output on my system

▶ Checking for latest commit ...
skipped

▶ Checking minimum requirements ...
Found Docker version 26.1.1
Found Docker Compose version 2.27.0

▶ Turning things off ...

▶ Creating volumes for persistent storage ...
Created sentry-clickhouse.
Created sentry-data.
Created sentry-kafka.
Created sentry-postgres.
Created sentry-redis.
Created sentry-symbolicator.
Created sentry-zookeeper.

▶ Ensuring files from examples ...
sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
symbolicator/config.yml already exists, skipped creation.

▶ Checking memcached backend ...
MemcachedCache found in sentry/sentry.conf.py, you should switch to PyMemcacheCache.
See:
  https://develop.sentry.dev/self-hosted/releases/#breaking-changes

[inoa-jboliveira]

Hey, it sounds like this is not a warning but an actual error. You guys need a better way to convey this information...
"you should" -> "you must" and something like "install.sh finished with errors!" :(

When I changed memcached, the upgrade did work.

I also realized that my sentry.conf.py is never being updated with latest sentry.conf.example.py. Any ideas on how to have these in sync? or how should it be installed differently? I wasnt the one configuring this server for the 1st time, so I don't know if having this conf file different from the example is something that just happens or if someone actively did it.

[azaslavsky]

The reason this was a warning was because we don't know how folks have modified their setups post-install. Saying "must" implies we know it's definitely true for your setup, when that is not necessarily the case.

I also realized that my sentry.conf.py is never being updated with latest sentry.conf.example.py. Any ideas on how to have these in sync? or how should it be installed differently? I wasnt the one configuring this server for the 1st time, so I don't know if having this conf file different from the example is something that just happens or if someone actively did it.

We avoid overwriting this for the same reason - people modify these files for various install-specific reasons all the time, and cogently syncing them in all such cases is a very hard problem. We've talked about moving to an "inheritance" setup where there is a sentry.default.conf.py that your file could overwrite, but we don't have that sort of thing on the road map or anything.

For now, we check for things that may need updating in the install script and surface messages to users at that time, making the assumption that they know their setups best.

[enricolucakindle]

I tried today these steps:

1. docker system prune -a
2. docker compose down
3. git status
4. git fetch
5. git checkout
6. sudo ./install.sh

There was no error anymore running the install script, but when i started the containers (docker compose up -d), there was another problem:

[hubertdeng123]

Those errors should not be happening when you're trying to bring your docker containers up. It indicates to me that your docker containers have not been properly built before running docker compose up -d. Could you try running the install script without sudo?

[hubertdeng123]

Just to be sure, is your sentry.conf.py specifying PyMemcacheCache as the backend?