Upgrade instructions

[SamGoody]

Hi,

I am using the self hosted version, and would like to update to the latest version.

1. Is there any way to see what version I am on, and what upgrades are available?
2. Is there a way to back up the DB data before updating?
3. What is the command to update.

I am using caddy-gen for plausible, so my startup script is:

docker-compose -f docker-compose.yml -f reverse-proxy/docker-compose.caddy-gen.yml up --detach

I am not much of a Docker person, but searching how to upgrade containers tells me I just have to run docker-compose pull:

docker-compose pull 
docker-compose -f docker-compose.yml -f reverse-proxy/docker-compose.caddy-gen.yml up --detach

However, I did that and LOST ALL OF MY ANALYTICS!

Interestingly, enough - I dod not lose my accounts, I just lost the analytics data - it still has the users and account data.

I have gathered from other tutorials that in older versions of Docker I would have had to do the following, but understand this is no longer relevant - and is even scarier to run:

# Method 1:
sudo docker pull [docker_image]
sudo docker stop [container_id]
sudo docker rm [container_id]
sudo docker run 

Please advise!

[ukutaht]

Hmm, I've always used the following and it works fine:

$ docker-compose down --remove-orphans
$ docker-compose pull plausible
$ docker-compose -f docker-compose.yml -f reverse-proxy/docker-compose.caddy-gen.yml up -d

It's weird that you lost your analytics data. Our docker-compose file stores the data in an internal docker volume called event-data: https://github.com/plausible/hosting/blob/master/docker-compose.yml#L19

Do you have that entry in your docker-compose? Do you see anything with event_data in your docker volume ls?

[Vigasaurus]

Yeah that's fair - and honestly, I haven't seen taking live backups specifically for Clickhouse be that problematic (I know you shouldn't but I've been doing it for the clickhouse instance here monthly for the past year without issue and a couple of successful restores).

One question I have for you there though @ukutaht, on the cloud version, how do you handle downtime during the image boot time (for example)? Do you have some middleware outside of the container in place to queue up requests that can't resolve in that time and relay them once the container is up? Or do you just let the events in that short time period fall off?

[ukutaht]

Currently we just have downtime when it happens. I'm planning to add a message queue (Kafka) to the infrastructure so we can keep accepting events while upgrading Clickhouse or something like that. The longer-term plan is to have multiple redundant Clickhouse instances so I can upgrade them one at a time.

[SamGoody]

Thanks.

I have added the update instructions to the docs as a pull request - the tiny little bit of help that I can offer.

Will look into backing up as suggested by Vigasaurus.

Again, thanks for making Plausible, and being so responsive even on the self-hosted option!

[SamGoody]

@Vigasaurus Am trying to use the backup script, but am a little too inexperienced with Docker :/
I realize this is not a Plausible thing, but can you direct me a little?

I use docker-compose, and the the docker_compose.yml file has the following volumes.

  db-data:
    driver: local
  event-data:
    driver: local
  geoip:
    driver: local

I need the volume to run the backup script, but I don't quite follow how the volumes work.
docker ps -a --filter volume= gives me an empty set for every volume name I can think of - db-data, event-data, plausible, hosting_plausible_1

Can you give me an example of how how I would do the backup?

[Vigasaurus]

Sure!

Firstly, you'll want to use docker volume ls to get the full names of the volumes, they might be plausiblehosting_event-data or similar, and not just the unprefixed name as from the docker-compose. Then, if you get an empty set (or containers that are stopped) for the volume filtered ps you posted (with the names from the command above), that's a good thing - it means no containers are actively using the volume so it's fully safe to backup.

Then, you'll want to use the following command to backup (you may or may not need sudo, it depends)

docker run -v VOLUME_NAME:/volume --rm loomchild/volume-backup backup - > ./BACKUP_FILE.tar.bz2 

or for example,

docker run -v plausiblehosting_event_data:/volume --rm loomchild/volume-backup backup - > ./clickhouse-2021_11_11.tar.bz2 

Then to restore from said file (as needed),

cat BACKUP_FILE.tar.bz2 | sudo docker run -i -v VOLUME_NAME:/volume --rm loomchild/volume-backup restore -

or for example,

cat ./clickhouse-2021-11-11.tar.bz2 | sudo docker run -i -v plausiblehosting_event-data:/volume --rm loomchild/volume-backup restore -

You could then use the same format with a different filename and volume name for the postgres volume as needed too.

[devrsi0n]

@ukutaht I also want to know how to get the version of a plausible docker image. I'm running a plausible instance and plausible frontend code for Chirpy, but there are some issues caused by server & client version mismatching. Could you guide me on this, thanks 🙏

[convernaticspeter]

@ukutaht
I have the same issue as the thread opener - after the update to the newest version all data is lost and the clickhouse DB isn't initialized.
I checked the data and it looks like the folder /var/lib/clickhouse, which stores the data for the volume was completely deleted during updating. What happened here?
I think it has to do with the update of Clickhouse to a newer version? But why is the data getting deleted

However inside the folder /var/lib/docker/volumes/hosting_event-data/_data/data/plausible_events_db there is 4 files that catched my attention:
events
events_v2 (created at the time the update was done)
sessions
sessions_v2 (created at the time the update was done)

I checked and the data is still in the container, but Clickhouse isn't using it. This seems to be an issue with Clickhouse.

[ruslandoga]

@convernaticspeter note that when updating to v2 you need to perform a data migration.

[ruslandoga]

More info in the release discussions:

• v2.0.0-rc.2 #3011
• v2.0.0-rc.1 #2981
• v2.0.0-rc.0 #2896

[Deliaz]

I just faced the same problem. Updated plausible via simple docker pull ... and all of the analytics data is gone.

[Deliaz]

@ruslandoga The migration guide helped. Thank you!
I wish it was on the website, as it was the first place I tried to search for instructions.

Anyway, thanks!

[chasmic]

I hit the same problem after upgrading "as usual", thinking that all data had been lost. Migrating the data was very straight forward once I was aware of the need and followed the database migration instructions in the github discussion.

Basically, this was the command I needed to run...
docker compose exec plausible bin/plausible rpc Plausible.DataMigration.NumericIDs.run

It would be good if the self-hosted documentation could give a heads up about this - it really is quite a heard stopping moment when you see lots of zeros after what appears to be a successful upgrade!

[ruslandoga]

👋 @chasmic

To better understand how to communicate breaking changes in releases, could you please give a brief outline of how you went about with this upgrade? Where did you learn about the new version, what other resources you visited, etc.?

[chasmic]

Absolutely! I have a monthly reminder to upgrade packages on the Docker server used for Plausible (apt-get update, upgrade, etc)
Then I would usually do the following after that... usually there's nothing to pull.

cd /opt/plausible/
docker-compose down --remove-orphans
docker-compose pull plausible
docker-compose up -d

All the previous releases (since this Plausible instance was set up) have been minor ones, so it hasn't been a problem pulling the latest version of Plausible (if there was one) "blind". To be honest, it's not been easy to notice that a release was available.

Now, after upgrading, I saw that there is a link to the releases page from the self-hosted documentation, and that this page has been updated by you a number of times since I last stumbled upon it! The release notes/instructions that you have written the past few months are much more informative than those for earlier releases.

That said, it would be very useful if the availability of a new release was communicated in the Plausible interface - the Github pages aren't somewhere I check regularly (unless I have a question/issue).

[ruslandoga]

Thank you!

the availability of a new release was communicated in the Plausible interface

This is probably the way to go. I'll ask the core team if they are open to that idea.

[antonreshetov]

The same problem, I just followed the commands that were in the documentation.

docker-compose down --remove-orphans
docker-compose pull plausible
docker-compose up -d

Now I have lost all my data. I am very upset.

[javedkumail]

root@analytics:/opt/plausible# docker compose exec plausible_events_db clickhouse-client
ClickHouse client version 22.6.9.11 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 22.6.9 revision 54455.

Warnings:
 * Linux is not using a fast TSC clock source. Performance can be degraded. Check /sys/devices/system/clocksource/clocksource0/current_clocksource
 * Linux threads max count is too low. Check /proc/sys/kernel/threads-max
 * Available memory at server startup is too low (2GiB).

bfe7066d6068 :) \c plausible_events_db

USE plausible_events_db

Query id: 7b2bc2f1-fd9f-4d14-847f-bfbd43e386e2

Ok.

0 rows in set. Elapsed: 0.003 sec.

fe7066d6068 :) select count(*) from sessions;

SELECT count(*)
FROM sessions

Query id: 8d960f2f-8768-4487-bcad-76a63b78f287


0 rows in set. Elapsed: 0.008 sec.

Received exception from server (version 22.6.9):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Table plausible_events_db.sessions doesn't exist. (UNKNOWN_TABLE)

bfe7066d6068 :) select count(*) from events;

SELECT count(*)
FROM events

Query id: 2648eb13-9562-45ce-a653-0733753644eb


0 rows in set. Elapsed: 0.004 sec.

Received exception from server (version 22.6.9):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Table plausible_events_db.events doesn't exist. (UNKNOWN_TABLE)

[ruslandoga]

Seems like neither events nor sessions tables exists.

Have you dropped them by any chance? Have you performed the Drop v1 tables (optional) step in #3132?

[javedkumail]

no i have not dropped v1 tables...i just performed steps as specified in step #3132

[javedkumail]

is there any other way i could retrieve data and find my previous data, it would be very helpful

[ruslandoga]

@javedkumail if the tables were dropped, the data might still be in some old unmerged parts. If something else happened -- I don't know. Either way, it might be better to ask ClickHouse team.

[benjaminbellamy]

Same here.
I just followed the doc https://plausible.io/docs/self-hosting#updating-plausible
I lost all my data.
:-/

[ruslandoga]

👋 @benjaminbellamy

You probably haven't lost your data, it's just not shown right now. Please see #3132 for how to finish the upgrade to v2.

[benjaminbellamy]

Indeed they were not lost but I was not able to get them alone. Doc should probably be updated.