1.23.8 -- crash in fetching notifications

[AdamMajer]

Description

It looks like the issue is null here somehow

journalctl
==========
Mai 15 14:27:31 src gitea[24343]: 2025/05/15 14:27:31 ...eb/routing/logger.go:78:func1() [W] router: failed    GET /api/v1/notifications?all=true&limit=20&page=1&status-types=read&subject-type=Pull for [2a01:4a0:11::2]:42722, panic in 14.7ms @ notify/user.go:18(notify.ListNotifications), err=runtime error: invalid memory address or nil pointer dereference


Mai 15 14:27:31 src gitea[24343]: 2025/05/15 14:27:31 ...rs/common/errpage.go:26:RenderPanicErrorPage() [E] PANIC: runtime error: invalid memory address or nil pointer dereference
Mai 15 14:27:31 src gitea[24343]: runtime/panic.go:792 (0x55b3b1ca09b1)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/routing/logger_manager.go:116 (0x55b3b37cb704)
Mai 15 14:27:31 src gitea[24343]: runtime/panic.go:792 (0x55b3b1ca09b1)
Mai 15 14:27:31 src gitea[24343]: runtime/panic.go:262 (0x55b3b1ca30b8)
Mai 15 14:27:31 src gitea[24343]: runtime/signal_unix.go:925 (0x55b3b1ca3088)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/models/issues/issue.go:192 (0x55b3b32514dc)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/models/issues/comment.go:469 (0x55b3b32438f2)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/services/convert/notification.go:60 (0x55b3b38ac9db)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/services/convert/notification.go:95 (0x55b3b38ad0ab)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/routers/api/v1/notify/user.go:90 (0x55b3b41bd015)
Mai 15 14:27:31 src gitea[24343]: reflect/value.go:584 (0x55b3b1d2f5c5)
Mai 15 14:27:31 src gitea[24343]: reflect/value.go:368 (0x55b3b1d2e6d8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:172 (0x55b3b37d5129)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/chain.go:31 (0x55b3b37cbfe5)
Mai 15 14:27:31 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:459 (0x55b3b37cef81)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/services/context/api.go:238 (0x55b3b41e903a)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/routers/api/v1/api.go:1666 (0x55b3b41e8bd6)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:73 (0x55b3b37ccb8e)
Mai 15 14:27:31 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:327 (0x55b3b37ce51a)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:459 (0x55b3b37cef81)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/services/context/access_log.go:73 (0x55b3b387c775)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/routing/logger_manager.go:122 (0x55b3b37cb58e)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: github.com/chi-middleware/proxy@v1.1.1/middleware.go:37 (0x55b3b414daf3)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/routers/common/middleware.go:59 (0x55b3b414eccf)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/routers/common/middleware.go:50 (0x55b3b414e85b)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/routers/common/middleware.go:36 (0x55b3b414e489)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:27:31 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:90 (0x55b3b37ccb4d)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/route.go:237 (0x55b3b37d709b)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/modules/web/route.go:175 (0x55b3b37d6a44)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:3301 (0x55b3b21914ad)
Mai 15 14:27:31 src gitea[24343]: net/http/server.go:3974 (0x55b3b2177230)
Mai 15 14:27:31 src gitea[24343]: net/http/h2_bundle.go:6529 (0x55b3b2140414)
Mai 15 14:27:31 src gitea[24343]: runtime/asm_amd64.s:1700 (0x55b3b1ca9100)

Gitea Version

1.23.8

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

self-compiled

Database

PostgreSQL

[wxiaoguang]

Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/models/issues/issue.go:192 (0x55b3b32514dc)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/models/issues/comment.go:469 (0x55b3b32438f2)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/services/convert/notification.go:60 (0x55b3b38ac9db)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/services/convert/notification.go:95 (0x55b3b38ad0ab)
Mai 15 14:27:31 src gitea[24343]: code.gitea.io/gitea/routers/api/v1/notify/user.go:90 (0x55b3b41bd015)

Code line doesn't match?

https://github.com/go-gitea/gitea/blob/v1.23.8/models/issues/comment.go#L469

[AdamMajer]

I've checked this already vs. our slightly patched package, and it looks like the same mismatch... 🤷

[wxiaoguang]

Well, the stacktrace should match ....

Is it possible that the panic is caused by your patch? 🤔

According to the code, the crash is caused by c.Issue.LoadRepo which accesses c.Issue.Xxx, but it is only called when there is no error after err := c.LoadIssue(ctx), in this case, the c.Issue can't be nil.

[AdamMajer]

No, panic is not caused by our patches.. They are LFS related in completely different part of the code...

There's another crash that has to do with deleting comments. It seems also related to an issue+comment combo here.

Mai 15 14:17:33 src gitea[24343]: 2025/05/15 14:17:33 ...eb/routing/logger.go:78:func1() [W] router: failed    POST /suse-edge/Factory/comments/15537/delete for 130.204.76.187:61144, panic in 20.2ms @ repo/issue_comment.go:289(repo.DeleteComment), err=runtime error: invalid memory address or nil pointer dereference
Mai 15 14:17:33 src gitea[24343]: 2025/05/15 14:17:33 ...rs/common/errpage.go:26:RenderPanicErrorPage() [E] PANIC: runtime error: invalid memory address or nil pointer dereference
Mai 15 14:17:33 src gitea[24343]: runtime/panic.go:792 (0x55b3b1ca09b1)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/routing/logger_manager.go:116 (0x55b3b37cb704)
Mai 15 14:17:33 src gitea[24343]: runtime/panic.go:792 (0x55b3b1ca09b1)
Mai 15 14:17:33 src gitea[24343]: runtime/panic.go:262 (0x55b3b1ca30b8)
Mai 15 14:17:33 src gitea[24343]: runtime/signal_unix.go:925 (0x55b3b1ca3088)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/routers/web/repo/issue_comment.go:301 (0x55b3b4261032)
Mai 15 14:17:33 src gitea[24343]: reflect/value.go:584 (0x55b3b1d2f5c5)
Mai 15 14:17:33 src gitea[24343]: reflect/value.go:368 (0x55b3b1d2e6d8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:172 (0x55b3b37d5129)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/chain.go:31 (0x55b3b37cbfe5)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:459 (0x55b3b37cef81)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/middleware/get_head.go:37 (0x55b3b41fceaa)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:182 (0x55b3b37d51c1)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/services/context/context.go:221 (0x55b3b3884404)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: gitea.com/go-chi/session@v0.0.0-20240316035857-16768d98ec96/session.go:257 (0x55b3b3778f4d)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:73 (0x55b3b37ccb8e)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:327 (0x55b3b37ce51a)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:459 (0x55b3b37cef81)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:73 (0x55b3b37ccb8e)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:327 (0x55b3b37ce51a)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:459 (0x55b3b37cef81)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/services/context/access_log.go:73 (0x55b3b387c775)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/routing/logger_manager.go:122 (0x55b3b37cb58e)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/chi-middleware/proxy@v1.1.1/middleware.go:37 (0x55b3b414daf3)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/routers/common/middleware.go:59 (0x55b3b414eccf)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/routers/common/middleware.go:50 (0x55b3b414e85b)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/routers/common/middleware.go:36 (0x55b3b414e489)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/handler.go:136 (0x55b3b37d55d9)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:2294 (0x55b3b21709e8)
Mai 15 14:17:33 src gitea[24343]: github.com/go-chi/chi/v5@v5.1.0/mux.go:90 (0x55b3b37ccb4d)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/route.go:237 (0x55b3b37d709b)
Mai 15 14:17:33 src gitea[24343]: code.gitea.io/gitea/modules/web/route.go:175 (0x55b3b37d6a44)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:3301 (0x55b3b21914ad)
Mai 15 14:17:33 src gitea[24343]: net/http/server.go:3974 (0x55b3b2177230)
Mai 15 14:17:33 src gitea[24343]: net/http/h2_bundle.go:6529 (0x55b3b2140414)
Mai 15 14:17:33 src gitea[24343]: runtime/asm_amd64.s:1700 (0x55b3b1ca9100)

[wxiaoguang]

They are the same phenomenons

err := xxx.LoadIssue()
if err != nil {
    return
}
access xxx.Issue.yyy // when execute to here, xxx.Issue can't be nil, but in your stacktrace, it seems to be nil

[AdamMajer]

The codepath looks familiar between the two crashes,

        if err := comment.LoadIssue(ctx); err != nil {
                ctx.NotFoundOrServerError("LoadIssue", issues_model.IsErrIssueNotExist, err)
                return
        }

        if comment.Issue.RepoID != ctx.Repo.Repository.ID {     <<<--- 
                ctx.NotFound("CompareRepoID", issues_model.ErrCommentNotExist{})
                return
        }

[AdamMajer]

Also happens after comment.LoadIssue() call

[wxiaoguang]

TBH I don't understand why it happens ... and why the code line in stacktrace doesn't match. Some thoughts (just FYI):

• Is it possible to figure out why the code line doesn't match? Indeed it doesn't seem right, and maybe the root cause is related.
• Is it possible to stably reproduce? If yes, good ... add more logs to see what happens.
• Is it possible to downgrade to an old version? If old version doesn't panic .... see the diff?

[AdamMajer]

So, LoadIssue() doesn't load the issue if the issue != nil. Could it be something else than nil but invalid somehow?

I'll look what's causing this more carefully. It's one of these "shouldn't happen but happens".

[wxiaoguang]

So, LoadIssue() doesn't load the issue if the issue != nil. Could it be something else than nil but invalid somehow?

Golang only uses "nil" or "non-nil" for struct pointer, no other "invalid value"