v2.0.0-rc.2

[ruslandoga]

The highlights of this release are:

• more visitor metrics like total visits and views per visit
• period comparisons [docs] [example]
• ability to change domains names Allow domain change #2803
• automatic geolocation database downloads replace geolix with locus #2362
• more email adapters support (Mailgun, Mandrill, SendGrid) Support alternative mailing services (Mailgun, Mandrill, Sendgrid) #2649

Upgrading Plausible Analytics to v2.0

⚠️ ⚠️ ⚠️

Upgrading to v2.0 requires performing a data migration.
Please read these notes until the end before deploying v2.0.0-rc.2

Ensure you are using a new ClickHouse version

The steps below have been tested with clickhouse/clickhouse-server:22.6-alpine please make sure to upgrade ClickHouse to at least this version.

Here's the excerpt from v1.5 release discussion regarding the ClickHouse upgrade

---

In your docker-compose.yml update the image used for plausible_events_db to a newer ClickHouse version:

Upgrading ClickHouse to 22.6

plausible_events_db:
- image: yandex/clickhouse-server:21.3.2.5
+ image: clickhouse/clickhouse-server:22.6-alpine

Restart the container

$ docker compose stop plausible plausible_events_db
$ docker compose rm plausible_events_db
$ docker compose up -d

This will boot up the new version of ClickHouse.

Related PR: plausible/community-edition#45

---

Update image tag

In your docker-compose.yml update the image used for plausible to v2.0.0-rc.2

plausible:
- image: plausible/analytics:v1.5.1
+ image: plausible/analytics:v2.0.0-rc.2

and restart the container

$ cd hosting
$ docker compose stop plausible
$ docker compose rm plausible
$ docker compose up -d

This will boot up the new version of the app.

If you open the dashboards now, you wouldn't see any past metrics. This is expected as v2.0 uses the new events_v2 and sessions_v2 tables to store analytics data. We need to perform data migration to copy the data into the new tables.

Run data migration

⚠️ ⚠️ ⚠️

You need enough free disk space available for x2 of the current plausible_events_db's event-data volume size. You can use something like docker system df -v | grep hosting_event-data to check how much space the current volume is occupying.

Connect to the running plausible container and start the migration flow

$ cd hosting
$ docker compose exec plausible bin/plausible rpc Plausible.DataMigration.NumericIDs.run

You can attempt this migration multiple times unless you drop v1 tables.

Drop v1 tables (optional)

Once you verify the migration went well, the old tables can be dropped. It's easiest to use clickhouse-client for this

$ ch hosting
$ docker compose exec plausible_events_db clickhouse-client

:) \c plausible_events_db
:) show tables;
:) drop table events;
:) drop table sessions;

See https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings#max-table-size-to-drop for how to drop tables with more than 50GB of data.

Enable automatic MaxMind GeoLite2 updates (optional)

In your plausible-config.env set MAXMIND_LICENSE_KEY environment variable and get an automatically updated GeoLite2 City geolocation database. The database edition is configurable with MAXMIND_EDITION environment variable and defaults to GeoLite2-City.

- IP_GEOLOCATION_DB=...
+ MAXMIND_LICENSE_KEY=LNpsJCCKPis6XvBP
+ MAXMIND_EDITION=GeoLite2-City

Note that for the changes in plausible-config.env to propagate to the plausible, the container needs to be recreated:

$ docker compose stop plausible
$ docker compose rm plausible
$ docker compose up -d

Also note that using GeoLite2-City edition requires more RAM than using GeoLite2-Country.

Now you can remove any other volumes and services used to download, store, and update geolocation databases.

Changelog

Following changes have been made since v1.5:

Added

• Add support for with_imported=true in Stats API aggregate endpoint
• Ability to use '--' instead of '=' sign in the tagged-events classnames
• 'Last updated X seconds ago' info to 'current visitors' tooltips
• Add support for more Bamboo adapters, i.e. Bamboo.MailgunAdapter, Bamboo.MandrillAdapter, Bamboo.SendGridAdapter Support alternative mailing services (Mailgun, Mandrill, Sendgrid) #2649
• Ability to change domain for existing site (requires numeric IDs data migration, instructions will be provided separately) UI + API (PUT /api/v1/sites)
• Add LOG_FAILED_LOGIN_ATTEMPTS environment variable to enable failed login attempts logs add LOG_FAILED_LOGIN_ATTEMPTS #2936
• Add MAILER_NAME environment variable support add MAILER_NAME #2937
• Add MAILGUN_BASE_URI support for Bamboo.MailgunAdapter add MAILGUN_BASE_URI #2935
• Add a landing page for self-hosters add landing page #2989

Fixed

• Fix tracker bug - call callback function even when event is ignored
• Make goal-filtered CSV export return only unique_conversions timeseries in the 'visitors.csv' file
• Stop treating page filter as an entry page filter
• City report showing N/A instead of city names with imported data Reject unknown imported cities from queries #2675
• Empty values for Screen Size, OS and Browser are uniformly replaced with "(not set)"
• Fix more pageviews with session prop filter than with no filters
• Cascade delete sent_renewal_notifications table when user is deleted Cascade delete sent_renewal_notifications table when user is deleted #2549
• Show appropriate top-stat metric labels on the realtime dashboard when filtering by a goal
• Fix breakdown API pagination when using event metrics Fix breakdown API pagination when using event metrics #2562
• Automatically update all visible dashboard reports in the realtime view
• Connect via TLS when using HTTPS scheme in ClickHouse URL update ecto to include db url scheme #2570
• Add error message in case a transfer to an invited (but not joined) user is requested improve transfer ownership error message #2651
• Fix bug with showing property breakdown with a prop filter
• Fix bug when combining goal and prop filters Fix unknown bind meta #2654
• Fix broken favicons when domain includes a slash

Changed

• Treat page filter as entry page filter for bounce_rate
• Reject events with long URIs and data URIs Reject events with long URIs and data URIs #2536
• Always show direct traffic in sources reports Always show direct traffic in sources reports #2531
• Stop recording XX and T1 country codes Ignore unknown countries #2556
• Device type is now determined from the User-Agent instead of window.innerWidth Use user-agent instead of screen_width to get device type #2711
• Add padding by default to embedded dashboards so that shadows are not cut off Embedded dashboard fixes #2744
• Update the User Agents database (https://github.com/matomo-org/device-detector/releases/tag/6.1.1)

Removed

• Remove Firewall plug and IP_BLOCKLIST environment variable
• Remove the ability to collapse the main graph Remove the ability to collapse the main graph + transition bug fix #2627
• Remove custom_dimension_filter feature flag remove custom_dimension_filter feature and views_per_visit_metric flags #2996

---

This discussion was created from the release v2.0.0-rc.2.

[Olakeen]

Still + image: plausible/analytics:v2.0.0-rc.1 ? not + image: plausible/analytics:v2.0.0-rc.2 ?

[ruslandoga]

👋 @Olakeen

Oops, sorry! I forgot to update the image tags.

Thank you for telling me about this!

UPDATE: updated :)

[Alexander01998]

I get the following error when trying to run data migration:

--rpc-eval : RPC failed with reason :nodedown

I'm updating from v1.5.1, running on Ubuntu 22.04.2 LTS.

I saw a similar discussion in #2896 (comment) where the issue was caused by a RELEASE_DISTRIBUTION=none variable in the plausible-conf.env file. However, I don't have this variable in my env file and I still get the error. 😕

Any ideas?

[ruslandoga]

ran into the --rpc-eval : RPC failed with reason :nodedown error

If you were executing these commands fast, then this error might have been due to sleep 10 in command at docker-compose.yml#L31 Meaning, the node hadn't been started by that time yet.

during migration, I was asked to confirm to drop a bunch of tables, which I did

Were you asked to confirm creation of any tables? :)

Full log of the data migration flow would be helpful.

saw events and sessions, but no events_v2 or sessions_v2

Seems like the data migration didn't finish.

[Alexander01998]

Thanks for your help @ruslandoga. I've figured out what went wrong.

When I was running the data migration script for the first time, I entered y every time that it asked me to confirm something.

I figured this would be reasonable, as for example the docker compose rm plausible command also asks a [y/N] question and needs a y answer to work. But I now realize that the first two queries of the data migration script (drop-events-v2 and drop-sessions-v2) are supposed to be denied in order for the script to run properly.

I've now re-run the script and simply pressed enter on every prompt so that it would run with all of the default yes/no answers. This worked. Everything has now been migrated successfully (log output).

It would be great if the migration guide could be updated with a note about these confirmation prompts, as I'm sure I'm not the only one who would incorrectly assume that they should be answered with y.

[ruslandoga]

docker compose rm plausible command also asks a [y/N] question and needs a y answer to work

I might be missing the point, but isn't it the same with the data migration script?

Entering y<enter> for DROP TABLE ... [y/N] drops the tables, <enter> or n<enter> -- it doesn't.

the first two queries of the data migration script (drop-events-v2 and drop-sessions-v2) are supposed to be denied in order for the script to run properly

I'm not sure that's true, I've tried it both ways y / n (drop, no drop) and it worked both times, since these dropped tables are created at a later step: https://gist.github.com/Alexander01998/600fdb6c4015c13cd721ce7e64b05c13#file-plausible_data_migration-log-L50-L89, -- if the tables already exist, that step is no-op, if the tables have been dropped, they are created anew.

t would be great if the migration guide could be updated with a note about these confirmation prompts, as I'm sure I'm not the only one who would incorrectly assume that they should be answered with y.

What should we write there?

[Alexander01998]

Huh, interesting. Every time I tried it with y, it didn't work. I'm pretty sure it never created the tables when I did that.

Since I still have my snapshot, let me run it again and get a log output for you...

[Alexander01998]

I can't reproduce the issue anymore. It just works now. https://gist.github.com/Alexander01998/eb9d5ffac55edf81025cd39ad1f391a4

[sgbx]

Sorry for the stupid question, I do not have much clue of docker. What/where exactly is the "hosting" folder?

[ruslandoga]

@pagenstedt no worries! The initial setup is described in https://plausible.io/docs/self-hosting

The hosting folder is just what I call the folder where the self hosting repo is cloned: git clone https://github.com/plausible/hosting

[sgbx]

OK, I see. Thank you.

[sgbx]

It was another issue. Docker Compose plugin was not installed. Now everything went fine.

[Eyesonly88]

What is the recommended and minimum CPU and RAM for this 2.0 version?

[ruslandoga]

👋 @Eyesonly88

I think it would be the same as before, https://plausible.io/docs/self-hosting#requirements :)

[ruslandoga]

v2.0.0 has been released: #3132

There are no significant changes from v2.0.0-rc.2 so it should be safe to update.

[ruslandoga]

Changes: v2.0.0-rc.2...v2.0.0

[chunkydotdev]

Upgraded using the instructions from the docs:

$ docker-compose down --remove-orphans
$ docker-compose pull plausible
$ docker-compose up -d

And thought I had lost all data, I really had a hallelujah-moment when I found this page! 😅

Please add the migration to the upgrade instructions-page so other people don't think they've deleted all their data after an upgrade. 🙏

[ruslandoga]

@Storken 👋

Right, the docs need to be updated. Maybe we can just link the discussions from there since they contain the most up to date info.

Thank you!

[ruslandoga]

I've added this note in plausible/docs#399