SQLITE_BUSY Messages

[tanknet203]

⚠️ Please verify that your question has not already been reported

• I have searched the existing issues and found no similar reports.

🛡️ Security Policy

• I have read and agree to Uptime Kuma's Security Policy.

📝 Describe your problem

I am getting random alerts sent from Uptime Kuma saying the monitored site is down when it isn't with the error message "SQLITE_BUSY: database is locked". By the next check the monitor shows back online.

DB is set to only hold 30 days of events, checks are every 60 seconds monitoring 53 entities (http, docker, ping)

📝 Error Message(s) or Log

🐻 Uptime-Kuma Version

2.3.0

💻 Operating System and Arch

Docker on UnRAID 7.2.4

🌐 Browser

Brave 1.88.138 (Chromium 146.0.7680.178

🖥️ Deployment Environment

• Runtime Environment:
• Database:
  • SQLite: Embedded
  • MariaDB: Embedded
• Database Storage:
  • Filesystem:
    • Linux: ZFS
  • Storage Medium: NVMe
• Uptime Kuma Setup:
  • Number of monitors: 53

[CommanderStorm]

We previously only got this from people running on verrry slow storage and merged an environment variable to correct for those.
Not sure why this is happening too for your nvme.

Can you post what the disk speed is via running one of the benchmarks?
(to avoid debugging something that is accidentally NFS or something..)

[tanknet203]

Yeah disk speed is definitely not the issue. i've seen throughput upwards of 600 MB/s on this setup without breaking a sweat. and I've never had this issue before upgrading this morning to 2.3.0

[ahembree]

I'm having the same issue with a much smaller instance (21 monitors) after the update to 2.3.0 this morning, I'm running within Docker on a Synology (not through their Container Manager) with the following disk speed on BTRFS I think:

# sudo dd bs=1M count=4096 if=/dev/zero of=/volume1/docker/testfile conv=fdatasync

4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.435 s, 376 MB/s

Trace: [Error: SQLITE_BUSY: database is locked] {
  errno: 5,
  code: 'SQLITE_BUSY'
}
    at process.unexpectedErrorHandler (/app/server/server.js:1993:13)
    at process.emit (node:events:519:28)
    at emitUnhandledRejection (node:internal/process/promises:252:13)
    at throwUnhandledRejectionsMode (node:internal/process/promises:388:19)
    at processPromiseRejections (node:internal/process/promises:475:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: [Error: update `stat_daily` set `up` = 35, `down` = 1, `ping` = 113.94285714285714, `ping_min` = 101, `ping_max` = 139 where `id` = 1903 - SQLITE_BUSY: database is locked] {
  errno: 5,
  code: 'SQLITE_BUSY'
}
    at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:1130:25)

[louislam]

I actually got this error too yesterday.

I am so confused, because the change in 2.3.0 didn't seem to affect anything without set the env var to true.
https://github.com/louislam/uptime-kuma/pull/7312/changes

Now, I am just not sure if I should merge #7307 (increase busy timeout), or revert #6979 (revert back to single connection).

[louislam]

So I read knex/knex#3176 (comment) again.

It seems that this comment is not fully true, since we have set to WAL mode, and just seeing some busy error.

Also:

As tsondergaard mentioned in the comments, even with PRAGMA busy_timeout, SQLite with multiple connections behaves quite differently from other databases - it will fail with SQLITE_BUSY when hitting a locked file rather than queuing the request.

So #7307 probably will not fully fix the issue. I think reverting #6979 may be the only solution sadly, since single connection is fine for many people, many years (but slow!).

[CommanderStorm]

I would say lets try merging #7307
There is a hope that this might imporve perf, so maybe we can ceep this 🤞🏻

[bt90]

@louislam sqlite can perfectly handle multiple read connections. But it shits the bed if you're treating it like PostgreSQL and attempt to do parallel writes.

The ideal solution would be a split of the pool. One for reading with X connections and one for writing with a single connection.

[bt90]

https://emschwartz.me/psa-your-sqlite-connection-pool-might-be-ruining-your-write-performance/

[louislam]

https://emschwartz.me/psa-your-sqlite-connection-pool-might-be-ruining-your-write-performance/

That's a good write up. I wish I could know this at the beginning of this project. Now I just don't know how to split a write-only connection for the whole project...

I would say lets try merging #7307
There is a hope that this might imporve perf, so maybe we can ceep this 🤞🏻

Yes, I think we try this first.

[otbutz]

Now I just don't know how to split a write-only connection for the whole project...

Can't we just run with two separate pools?

[corysell]

I just realized last night that I was running a very old version. The upgrade process took around 17 total hours, but when it finally finished - I am seeing these exact same errors.

I see something was merged, I'm tagged :2, so I believe I'm on the most recent. Let me know if there's anything I can provide or test for a fix!

I'm running my containers on an NVME, so storage speed should not be a concern.

[DPattee]

I updated my uptime-kuma container today and now I am getting these errors on a system that was working perfectly fine last week. I can't even log in to Kuma now - after entering username and password the web interface just locks up. Reloading the web page and entering an deliberately incorrect username / password gets it to show the 'incorrect...' message, then entering the correct & clicking 'log in' locks it up again.

Underlying system is perfectly fine - it is running on a synology with plenty of high speed storage, and no memory or cpu pressure.

From the logs, at least some of the monitors are running because I see some warnings about failures.

But I can't complete login and now 50+ monitors I've configured over the past week are apparently held hostage

update: I've had to just shut kuma down. It has started to ping in our team chat that there are service outages, but the error message attached to the chat message is "Error SQLITE_BUSY: database is locked"