Three hours of data missing from the day's report

[kristjanmar]

Past Issues Searched

• I have searched open and closed issues to make sure that the bug has not yet been reported

Issue is a Bug Report

• This is a bug report and not a feature request, nor asking for self-hosted support

Using official Plausible Cloud hosting or self-hosting?

Plausible Cloud from plausible.io

Describe the bug

My daily data for Sunday 26-Jun-2022 is missing three hours of data between 4-8 pm. I have several uptime checkers and they do not report the site going down for more than a few minutes last night, so this is has to be an issue with Plausible.

I also don't see an unusual amount of error reports from my Plausible proxy config in Vercel that could explain such a huge drop (thousands of visits missing). I do have a few of these though:

502 [stockanalysis.com/stats/api/event/](http://stockanalysis.com/stats/api/event/)
DNS_HOSTNAME_RESOLVE_FAILED: An error occurred with this application.

Expected behavior

This should not happen.

Screenshots

You can see that the chart goes to near-zero for one hour and absolute zero for two hours.

Environment

- OS: Mac OS
- Browser: Chrome, latest

[kristjanmar]

Also, the numbers in the report don't match. The overview page for example shows Google traffic at 6.2K, but when I click to filter "Source is Google" then it shows 5.2K. The other sources are also wrong.

[metmarkosaric]

thanks for reporting this @kristjanmar and sorry for the inconvenience! our developers are looking into what happened and will share more as soon as possible.

[ukutaht]

Yes this issue was caused by our Clickhouse database being overloaded with INSERT statements for a few hours. I suspect it has to do with imports from google analytics.

Our monitoring failed to catch the issue. We're working to improve our monitoring so we can get alerted and fix issues like this quicker.

The inconsistency in Top Sources is caused by the database being down. Since our data is denormalized for faster querying, there can be errors during incidents. We could calculate everything from scratch every time but that would be much more expensive for CPU/RAM.

Currently the system is stable so I will close this issue. We're sorry about the incident and we're doing what we can to avoid issues like this in the future.

[vinodbollini]

@ukutaht will the traffic data from that period be lost? Or will it be accessible after all your database INSERTs/WRITEs have synced?
We have a couple of customers who claimed that the site was down, unfortunately at the same time as this. Wanted to compare traffic to see if there was any actual drop.

[ukutaht]

The writes from that period are gone. Unfortunately we cannot keep logs so we cannot restore it either.

We'll learn from this and make our systems more reliable. I'm sorry that the data was lost this time :(

[kristjanmar]

Looks like it's down again, both the dashboard and the tracking. I really like Plausible but frequent outages like this are going to be a deal breaker for me, unfortunately.

503 Service Unavailable
No server is available to handle this request.

[metmarkosaric]

sorry about that @kristjanmar! this is still related to the GA imports. it's a relatively new feature so we haven't encountered these issues before. everything is back running fine again. and again, sorry about the incident. we're doing what we can to avoid issues like this in the future!