Failed migration v67 remove stale watches causes Gitea to not start

[KubqoA]

• Gitea version (or commit ref): v8a9f5b3
• Git version: don't know, but most probably not related to the issue
• Operating system: Alpine Linux v3.8 running the Gitea's Docker container
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist: https://privatebin.net/?73672c209412b2f4#DlI7DnhdZH8DkSoFl7qPxXzYXOhAa8J+B9hp/voyrug=
• docker-compose.yml: https://privatebin.net/?58383a8ad08c6bd2#kMhn10MYdNrIR8vZtEIe1DXPdIal/ZTA53vrch3+vyU=
• app.ini: https://privatebin.net/?d9584bd98f475c6f#eGvSbTLT7VjAQdZxA7ccjZgCbGkiXfhE6NTtVLSH/AQ=

Description

I am running mailcow with gitea so I can authenticate users using their mail account. Yesterday I updated my mailcow stack and also Gitea with it. Since then Gitea doesn't launch. I tried to debug it and it looks like it gets stuck when it tries to run migration remove stale watches. If I specify another database file in my app.ini, so that Gitea creates a new one, everything seems to work and Gitea loads without any problems. However this means that I'd lose all my data, which is not a very good solution.

[lafriks]

Can you check contents of gitea.log file, it should contain error where it fails

[KubqoA]

I checked the /var/lib/gitea/gitea/logs/gitea.log, xorm.log and serv.log, but there are no other logs. The log gist I gave you has all the log messages I am able to get my hands on.

[zeripath]

Hmm is this an sqlite database? I wonder if you're hitting a deadlock somehow

[KubqoA]

Well, if migrating to different type of database is a possible solution I am down for it. However, I don't know would I do such a thing.

[typeless]

@KubqoA What was the version before the migration?

[zeripath]

The logs you've printed refer to migration v67.go. the final log appears to be due to a select on the repository table and is probably this line:

gitea/models/migrations/v67.go

Line 92 in d9b51a7

if _, err := x.Get(repo); err != nil {

Which probably shouldn't be x.Get but rather be sess.Get

[KubqoA]

So the issue is in that migration file?

[typeless]

It is the exact line @zeripath demonstrated.

[KubqoA]

So should I create a pull request so this can get fixed?

[zeripath]

Yup. You're hitting an SQL deadlock in the v67 migration.

I recently did a check up of the rest of the codebase looking for deadlocks but skipped the migrations coz I assumed incorrectly that these would have been caught already. Clearly I need to check them aswell.

In your case this should be a very easy fix - although I won't be able to make a PR for several hours possibly not till Saturday so if someone else could do it I can lgtm it. The simplest way to prevent deadlocks is to ensure that all dependent db queries are done within the same transaction.

(You can be a bit cleverer but it will almost always bite you later.)

[typeless]

@KubqoA That would be nice 😄

[KubqoA]

Created the pull request @zeripath

[NetOpWibby]

Is this why upgrading to 1.7.0 isn't working?

[zeripath]

@NetOperatorWibby without more information from you I couldn't possibly comment