Enable cron.git_gc_repos by default to avoid mirror syncing errors

[pboguslawski]

Description

Just noticed https://github.com/nextcloud/contacts mirror sync error:

2023/03/29 08:08:31 ...irror/mirror_pull.go:279:runSync() [E] [6423d5db-14] SyncMirrors [repo: 38:nc/nextcloud-contacts]: unable to write commit-graph for '/myreporoot/nc/nextcloud-contacts.git' : exit status 128 - error: Could not read 544c4227b061eb5331617430a3896872fd3915ab
2023-03-29    08:08:31    gitea[123]: error: Could not read a3a65b5a36972ed4c50d461e9fe6127a66c2ea05
2023-03-29    08:08:31    gitea[123]: error: Could not read 681e22178c7cefeb3033740f2cbb93312a615fc9
2023-03-29    08:08:31    gitea[123]: error: Could not read 24df40d4cf028ba93ff68f7230e059dce1374c2f
2023-03-29    08:08:31    gitea[123]: error: Could not read 82fb27f04da07758971fff6f9d99d30e21f61295
2023-03-29    08:08:31    gitea[123]: error: Could not read 46ae973a9d8c6abf9b07da4ce0a27014c275f459
[...]
2023-03-29    08:08:31    gitea[123]: error: Could not read e4b86229e61575ab45542e6d337ddf3892a78f4f
2023-03-29    08:08:31    gitea[123]: error: Could not read 03a21dd37f6f01da2d637f789baa4066271d1080
2023-03-29    08:08:31    gitea[123]: fatal: unable to parse commit 03a21dd37f6f01da2d637f789baa4066271d1080
2023-03-29    08:08:31    gitea[123]: - error: Could not read 544c4227b061eb5331617430a3896872fd3915ab
2023-03-29    08:08:31    gitea[123]: error: Could not read a3a65b5a36972ed4c50d461e9fe6127a66c2ea05
2023-03-29    08:08:31    gitea[123]: error: Could not read 681e22178c7cefeb3033740f2cbb93312a615fc9
[...]
2023-03-29    08:08:31    gitea[123]: fatal: unable to parse commit 03a21dd37f6f01da2d637f789baa4066271d1080
2023-03-29    08:08:31    gitea[123]: 

Mirror was created many months ago and worked fine (everyday sync) till today. Manual resync throws same error in this mirror.

Commit 544c4227b061eb5331617430a3896872fd3915ab mentioned in errors is present in github but not in mirror (looks like orphaned on github).

Fresh https://github.com/nextcloud/contacts mirror setup on other system with same gitea+git versions does not throw such error on sync.

Gitea Version

v1.18.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.30.2

Operating System

No response

How are you running Gitea?

Compiled from sources.

Database

MySQL

[wxiaoguang]

A quick google search says that git fsck or git gc might help. It looks like some git internal data corrupted (or out of sync)

[pboguslawski]

Running Garbage collect all repositories manually from Monitoring resolved the issue, thank you.

git gc is fired automatically only during some operations, so probably is not fired on mirror update and that caused the problem. Please consider enabling cron.git_gc_repos by default to avoid such surprises. gc and fsck jobs should probably have longer timeout by default (1800s maybe?) to handle bigger repos on slower hardware by default.

[lunny]

Running Garbage collect all repositories manually from Monitoring resolved the issue, thank you.

git gc is fired automatically only during some operations, so probably is not fired on mirror update and that caused the problem. Please consider enabling cron.git_gc_repos by default to avoid such surprises. gc and fsck jobs should probably have longer timeout by default (1800s maybe?) to handle bigger repos on slower hardware by default.

Or maybe we can change the issue's title to proposal a default value change?

[pboguslawski]

Done.

[adlternative]

Concurrent use of git gc and some git commands may lead to data corruption. I don't know if it is related to what is happening here: https://lore.kernel.org/git/CAOLTT8Tt3jW2yvm6BRU3yG+EvW1WG9wWFq6PuOcaHNNLQAaGjg@mail.gmail.com/

[pboguslawski]

Concurrent use of git gc and some git commands may lead to data corruption.

To avoid problems consider:

• adding warning to cron.git_gc_repos docs about corruption possibility with link to mentioned page,
• introducing checkbox in every repo config to explicitly allow cron.git_gc_repos to process it (similar to existing Enable Repository Health Checks (git fsck) checkbox).

With such checkboxes admin may enable cron.git_gc_repos only for repos they consider as non-critical and safe (i.e. read only mirrors).