Repo Mirror authentication failure creates signal 15 and shuts down gitea

[data-davey]

Description

I recently discovered after upgrading to Gitea 1.23.6 that an expired Github PAT which I use to authenticate when mirroring a gitea repo to gihub, eventually generates a Signal 15 and shuts down my gitea server.

Not sure if this happens with earlier versions.

I run this as a container using podman-compose. It's been incredibly stable until the PAT expired, and I upgraded. Not sure if the expired PAT on an earlier version will create the same shutdown scenario.

2025/03/31 05:58:51 ...irror/mirror_push.go:116:SyncPushMirror() [E] SyncPushMirror [mirror: 9][repo: <Repository 5:data-davey/sec-data-py>]: push failed: exit status 128 - remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/data-davey/sec-data-py/'

• remote: Invalid username or password.
  fatal: Authentication failed for 'https://github.com/data-davey/sec-data-py/'
• remote: Invalid username or password.
  fatal: Authentication failed for 'https://github.com/data-davey/sec-data-py/'

2025/03/31 06:32:37 ...eb/routing/logger.go:102:func1() [I] router: completed GET / for 10.89.0.3:43794, 200 OK in 23.8ms @ web/home.go:32(web.Home)
2025/03/31 06:32:38 ...eb/routing/logger.go:102:func1() [I] router: completed GET /repo/search?count_only=1&uid=1&team_id=undefined&q=&page=1&mode= for 10.89.0.3:43794, 200 OK in 4.7ms @ repo/repo.go:573(repo.SearchRepo)
2025/03/31 06:32:38 ...eb/routing/logger.go:102:func1() [I] router: completed GET /repo/search?sort=updated&order=desc&uid=1&team_id=undefined&q=&page=1&limit=15&mode=&archived=false for 10.89.0.3:43794, 200 OK in 7.6ms @ repo/repo.go:573(repo.SearchRepo)
2025/03/31 06:32:39 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/events for 10.89.0.3:43802, 200 OK in 1392.3ms @ events/events.go:18(events.Events)
2025/03/31 06:32:39 ...eb/routing/logger.go:102:func1() [I] router: completed GET /data-davey/sec-data-parser for 10.89.0.3:43794, 200 OK in 55.7ms @ repo/view_home.go:332(repo.Home)
2025/03/31 06:32:42 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/events for 10.89.0.3:43794, 200 OK in 2780.6ms @ events/events.go:18(events.Events)
Received signal 15; terminating.
2025/03/31 06:35:41 ...eful/manager_unix.go:144:handleSignals() [W] PID 13. Received SIGTERM. Shutting down...
2025/03/31 06:35:41 cmd/web.go:366:listen() [I] HTTP Listener: 0.0.0.0:3000 Closed
2025/03/31 06:35:41 ...eful/server_hooks.go:46:doShutdown() [I] PID: 13 Listener ([::]:3000) closed.
2025/03/31 06:35:41 .../graceful/manager.go:168:doHammerTime() [W] Setting Hammer condition
2025/03/31 06:35:42 .../graceful/manager.go:184:doTerminate() [W] Terminating
2025/03/31 06:35:42 ...er/issues/indexer.go:120:2() [I] PID: 13 Issue Indexer closed
2025/03/31 06:35:42 ...eful/manager_unix.go:154:handleSignals() [W] PID: 13. Background context for manager closed - context canceled - Shutting down...

Gitea Version

1.23.6

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

Debain Bookworm

How are you running Gitea?

Self hosted Gitea on a Raspberry Pi, running as a container with Podman.

Database

PostgreSQL

[data-davey]

As it turns out, the failed auth is not the cause. Seems to be sigterm coming from somewhere else. Further investigation needed.

[abjoseph]

@data-davey I'm seeing a similar behavior with my instance; I do a lot of mirroring as well. Did you ever identity a root cause?

[TheFox0x7]

@abjoseph @data-davey
I've encountered similar issue during feature testing when I gave the container fairly low memory limit and git process that was called by gitea took too much memory.
It might not be the same cause as here, but if you have insights into container memory consumption you can check if it got very high.

[abjoseph]

@TheFox0x7

Thanks for providing an additional data point on this, I have my LXC machine configured with 4 CPU and 3 GB RAM, the memory usage never went about 600MB however, the CPU did spike quickly to about 20 percent usage and then the Gitea process got the SIGTERM signal and shutdown. It is a weird behavior and happens infrequently but consistently enough as of late that I suspect it may be related to a recent update/release.

[TheFox0x7]

One thing you can also try is enabling trace logging and checking what happened before the signal was issued. I don't have idea what causes this beyond what I triggered on accident, but that was under VERY low ram (200MB) scenario.

[data-davey]

@abjoseph yes I discovered the root cause. I am running Gitea in a container using Podman. When you log out, podman containers are shutdown by systemd unless you enable-linger.

After I did this it works flawlessly.