datalad update fails randomly with error: "cannot lock ref 'refs/remotes/origin/master'" and ".... git-annex"

[mlell]

What is the problem?

Sometimes, I encounter errors like this (I replaced file paths by <$PWD> and <$REMOTE>. This is confusing for me and even more so for my more inexperienced colleagues. It prints that it executes git gc in the background. Could that be the reason for the problem?

16:56 $ datalad update --how ff-only --reobtain-data
update(error): . (dataset) [Fetch failed: CommandError(CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false fetch --verbose --progress --no-recurse-submodules --prune origin' failed with exitcode 1 under /<$PWD> [err: 'remote: Total 96161 (delta 36595), reused 86591 (delta 27875), pack-reused 0
error: cannot lock ref 'refs/remotes/origin/master': is at 676022296cec3c0f0a30c2c9ead36b39d220c798 but expected 7cad1f60410049dd7876e0c55c63bccde0ef17b4
From /<$REMOTE>
 ! 7cad1f6041..676022296c  master               -> origin/master  (unable to update local ref)
error: cannot lock ref 'refs/remotes/origin/git-annex': is at 86d9fdbc89826c0d5834c892c45bcb6c46b4c23e but expected 6b9ef460d253645fc2079430c6b561c2a4b89339
 ! 6b9ef460d2..86d9fdbc89  git-annex            -> origin/git-annex  (unable to update local ref)
 = [up to date]            lell-unfinished      -> origin/lell-unfinished
 * [new tag]               before-big-recompute -> before-big-recompute
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.'])] [CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false fetch --verbose --progress --no-recurse-submodules --prune origin' failed with exitcode 1 under<$PWD> [err: 'remote: Total 96161 (delta 36595), reused 86591 (delta 27875), pack-reused 0        
error: cannot lock ref 'refs/remotes/origin/master': is at 676022296cec3c0f0a30c2c9ead36b39d220c798 but expected 7cad1f60410049dd7876e0c55c63bccde0ef17b4
From <$REMOTE>
 ! 7cad1f6041..676022296c  master               -> origin/master  (unable to update local ref)
error: cannot lock ref 'refs/remotes/origin/git-annex': is at 86d9fdbc89826c0d5834c892c45bcb6c46b4c23e but expected 6b9ef460d253645fc2079430c6b561c2a4b89339
 ! 6b9ef460d2..86d9fdbc89  git-annex            -> origin/git-annex  (unable to update local ref)
 = [up to date]            lell-unfinished      -> origin/lell-unfinished
 * [new tag]               before-big-recompute -> before-big-recompute
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.']]

Executing the same command again does not fail but runs without errors.

What steps will reproduce the problem?

I do not know how to reproduce because it appears sometimes and executing the same command again works.

DataLad information

Datalad version: 0.19.3
git version: 2.40.1
git-annex version: 10.20230926-g44a7b4c9734adfda5912dd82c1aa97c615689f57
Rocky Linux 9.3

Additional context

No response

Have you had any success using DataLad before?

No response

[yarikoptic]

Congratulations, you unraveled some new (to me at least; with no hits on our organization) "aspect" somewhere deeper in the God's land of git. Google has a good number of hits e.g. https://timmousk.com/blog/git-unable-to-update-local-ref/ but here is

overall "AI summary" from chatgpt:

The "unable to update local ref" error in Git typically occurs due to a few common reasons. The details you provided point towards issues with local references that Git is unable to update during a fetch operation. Here are some potential causes and solutions for this error:

Causes

1. Locked References or Files: If a reference or a file in the .git directory is locked (due to another Git process running or an abrupt termination of a Git command), Git may not be able to update the local references during a fetch.

2. Corrupted Local References: The local reference might be pointing to a commit that Git does not expect, possibly due to a corruption in the local repository.

3. Non-Fast-Forward Changes: If there are changes in the remote repository that cannot be fast-forwarded to your local branch, you might encounter issues, though this typically manifests differently than the error message you're seeing.

Solutions

Check for Lock Files

First, ensure no ongoing Git operations are running. Check for any .lock files within the .git/refs/remotes/origin/ directory and remove them if no Git processes are running. These files can prevent Git from updating references.

find .git/ -name '*.lock'

If you find any .lock files and are sure there are no ongoing Git operations, you can safely remove them.

Repair Corrupted References

If a reference is corrupted or in an unexpected state, manually fixing or deleting the problematic reference file might solve the issue. However, be cautious with manual deletions, as they can affect repository integrity.

1. Backup the Refs: Before making changes, it might be wise to backup the refs directory.

2. Delete Problematic Refs: You can try removing the problematic reference files. Git will recreate them from the remote information on the next fetch.

   rm .git/refs/remotes/origin/master
   rm .git/refs/remotes/origin/git-annex

After removing these files, attempt to fetch again:

git fetch origin

Prune and Clean

Sometimes, simply pruning and cleaning up outdated references can resolve such issues:

git fetch --prune
git remote prune origin

Check for Repository Corruption

If the issue persists, there might be corruption within your repository. Running a consistency check can help identify and, in some cases, resolve issues:

git fsck --full

If git fsck reports issues, you might need to take further steps to repair the repository, which could involve more detailed commands based on the output of fsck.

Last Resort: Clone Anew

If none of the above solutions work, the most straightforward way to resolve this issue might be to clone the repository anew. Ensure you've pushed all local changes to remote or backed them up before doing so.

git clone <repository-url>

After cloning, you can manually apply any changes from your previous repository if needed.

The summary so far -- not sure it is really a DataLad issue -- might be more of some underlying git/filesystem etc interaction. Do direct invocations of git mentioned in the error work?

I would recommend updating git, sharing/looking into filesystem details, and also get newer DataLad of cause ;)