Pushing docker images using gitea actions/act_runner times out due to slow DB update

[LouisVallat]

Description

Often, when having multiple CI running using act_runner (on another server) and gitea actions, docker image uploads get stuck, returns a 504 Gateway time out (using an nginx reverse proxy), logs shows database updates sometimes running for 3 minutes, causing these timeout issues.
I changed hosts twice, nothing appears in gitea doctor nor in the gitea admin self check.

The issue seems to be caused by this:

Feb 26 20:34:55 gitea gitea[166172]: 2025/02/26 20:34:55 ...dules/actions/log.go:73:WriteLogs() [W] [Slow SQL Query] UPDATE dbfs_data SET revision=revision+1, blob_data=$1 WHERE meta_id=$2 AND blob_offset=$3 [[50 48 50 53 45 48 50 45 50 54 84 49 57 58 50 52 58 52 52 46 49 51 55 50 49 55 55 90 32 104 111 109 101 108 97 98 40 118 101 114 115 105 111 <TRUNCATED> 66 32 51 56 56 46 53 115 10] 1311 0] - 3m33.427483567s
Feb 26 20:34:55 gitea gitea[166172]: 2025/02/26 20:34:55 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/UpdateLog for [2001:<REDACTED>]:0, 500 Internal Server Error in 213484.3ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)

It looks like the update query took 3m33.427483567s, causing a 500 error (in a request that took 213484.3ms).
Could this happen due to an influx of AI scraper bots?

Gitea Version

1.23.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

git version 2.39.5

Operating System

Debian 12

How are you running Gitea?

I am running a manual install of Gitea (binary, systemd) on a Debian 12 server, with Redis and Postgres.
There is plenty of disk space, memory headroom and disk IOPS to spare, IOWAIT is kept under 1%.
The instance is setup behind a reverse proxy.

Database

PostgreSQL

[lunny]

How many records in the table dbfs_data?

[wxiaoguang]

What's the network between your Gitea instance and database?

It seems that the database can't finish the SQL UPDATE dbfs_data SET blob_data={dozens of kilobytes} in time.

I guess either:

• There is some network problem (packet loss, laggy, or firewall)
• Database is slow (need to use database utilities to figure out whether similar large UPDATE SQLs would lag)

[LouisVallat]

How many records in the table dbfs_data?

There are 22.

gitea=# SELECT COUNT(id) FROM dbfs_data;
 count
-------
    22
(1 row)

gitea=#

for a total of 512kB

gitea=# SELECT pg_size_pretty(pg_total_relation_size('dbfs_data'));
 pg_size_pretty
----------------
 512 kB
(1 row)

gitea=#

---

What's the network between your Gitea instance and database?

Postgres and Gitea are hosted on the same machine, gitea connects to its DB using localhost.

[wxiaoguang]

Maybe some database transactions or mutex/lock are abused ..... just a guess, I don't use act_runner and have no idea at the moment ......

[lunny]

Can you try connecting to the database from a shell and measure the operations speed?

[LouisVallat]

Can you try connecting to the database from a shell and measure the operations speed?

What do you mean by "measure the operations speed"? Run the update query manually in the DB with some timing measurement in place?

Tbh it doesn't happen all the time and seems to have mostly resolved itself now, except a certain delay between the end of a docker image upload and the final validation/confirmation but that may be expected). The gitea instance was being scrapped by AI crawlers at the time of the issue.

The delay I was talking about:

Feb 27 13:58:09: [task name]   | #8 pushing layer <hash> 1.14GB / 1.14GB 24.7s
Feb 27 13:58:32: [task name]   | #8 pushing layer <hash> 48.2s done