Description
What happened?
TL/DR:
During checkout, pruning tags before fetching them does not happen in certain circumstances.
If the new tags from the server conflict with local ones, build fails.
Scenario:
We had a tag, let's name it "one/two". One day we deleted it from the server and added a tag "one" instead.
git cannot have both tags at the same time, because they are held as file system objects under .git/refs/tags
, so one/two
existing would prevent one
from being created, and vice versa.
Nightly builds had already fetched the one/two
tag into agent workspace.
Normally, the --prune-tags
added by #3544 would delete the old tag from workspace before fetching the new one., but this did not happen during a PR build, see logs below.
My analysis:
added --prune-tags
to git fetch
options, this works as expected for scheduled builds, the removed tag gets deleted and the new one fetched:
git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin
From https://[redacted]
- [deleted] (none) -> one/two
- [new tag] one -> one
For PR builds however, git is given a ref wildcard to fetch: +refs/heads/*:refs/remotes/origin/* +refs/pull/<number>/merge:refs/remotes/pull/<number>/merge
This apparently leads git to skip pruning of tag refs, while still fetching new tag refs!
It might be a bug in git itself: Either providing ref wildcards should only affect refs matching this wildcard (so no tags would be fetched or deleted, ignoring both --tags
and --prune-tags
), or specifying --tags --prune-tags
should always fetch and delete tags, partially ignoring the ref wildcard.
I think the latter is better, but will break behavior in an unexpected way for many people... So I don't expect a bugfix from git.
Possible Solutions:
- in its error message, git suggests to do an explicit
git remote prune origin
before fetching. - when giving a ref wildcard to fetch, we may just need to include tags. Not sure that this will work, and may make the whole thing mor complicated.
- make a request for change of behavior in git itself would be possible but sounds inadvisable: As mentioned, starting to prune tags even if the explicit ref wildcard say not to, may be too much of a breaking change, so I assume if git fixes this, they will instead not fetch tags anymore. That would prevent causing this conflict, but also not give us the tags we may rely on. So presumably git won't change that behavior at all.
Versions
Azure DevOps Server 2022.2 (AzureDevopsServer_20250226.1)
Current agent version: '3.240.1'
git version 2.34.1
OS: Debian Linux container on Ubuntu Linux agent
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
Azure DevOps Server (Please specify exact version in the textbox below)
Azure DevOps Server Version (if applicable)
Azure DevOps Server 2022.2
Operation system
Debian Linux container on Ubuntu Linux agent
Version controll system
git version 2.34.1
Relevant log output
> git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/pull/<number>/merge:refs/remotes/pull/<number>/merge
> remote: Azure Repos
> remote:
> remote: Found 19 objects to send. (5 ms)
> From https://<redacted>
> <hash>..<hash> branch1 -> origin/branch1
> <hash>..<hash> branch2 -> origin/branch2
> * [new ref] refs/pull/<number>/merge -> pull/<number>/merge
> error: cannot lock ref 'refs/tags/one': 'refs/tags/one/two' exists; cannot create 'refs/tags/one'
> ! [new tag] one -> one (unable to update local ref)
> error: some local refs could not be updated; try running
> 'git remote prune origin' to remove any old, conflicting branches
> ##[warning]Git fetch failed with exit code 1, back off 3.095 seconds before retry.
>