fetch will give a fatal error once when a new arbitrary branch on the remote contains a submodule url change with updated sha

[martinv-github]

fetch will give a fatal error once when a new arbitrary branch on the remote contains a submodule url change with updated sha

Local repro steps (complete test script provided below)

• create repo A with a commit on master
• create repo B with a commit on master
• create repo X and add a submodule S1 pointing to A, commit to master.
• clone repo X with submodules to repo Z
• in repo X, create branch X1 from master. Change submodule S1 url to repo B and commit on X1.
• in repo Z, fetch and notice the fatal error "upload-pack: not our ref SHA"

From there, you can then quickly repro the error like so:

• in repo Z , delete branch origin/X1 and then fetch again.

Current workaround is to set submodule.recurse to false
I could not repro on Git for Linux 1.9 (which is the one bundled in my Linux Ubuntu subsystem for Windows)

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

git version 2.33.0.windows.2
cpu: x86_64
built from commit: 8735530946cced809cc6cc4c2ca3b078cdb3dfc8
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Microsoft Windows [Version 10.0.18363.1556]

• What options did you set as part of the installation? Or did you choose the
  defaults?

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: WinSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Core
Performance Tweaks FSCache: Enabled
Enable Symlinks: Enabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

NA

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

#!/bin/bash

die()
{
    echo "FATAL"
    exit 1
}

createRepo()
{

    local name=$1
    [[ -n "$name" ]] || die
    mkdir $name || die
    pushd $name
    git init
    touch readme.txt && git add readme.txt && git commit -m first
    popd
    echo "created git repo '$name'"
}

printHeader()
{
    echo "=============================================================="
    if [[ -n "$1" ]]; then
        echo "$1"
        echo "=============================================================="
    fi
}

main()
{
    local root=$(pwd)
    rm -rf child{1,2} user{1,2}
    #create submodules
    printHeader "setup submodules"
    createRepo child1
    createRepo child2
    #workaround for identical initial SHA
    pushd child2
    touch foobar && git add foobar && git commit -m second
    popd

    #create main repo
    printHeader "setup main repo"
    createRepo user1
    #add submodule
    git -C user1 submodule add -b master "${root}/child1" sub1
    git -C user1 commit -m 'added submodule'
    #another user clones repo
    printHeader "setup second user"
    git clone --recurse-submodules user1 user2

    #switch submodule url in main repo in arbitrary branch
    printHeader "switch submodule url"
    pushd user1
    git checkout -b exper
    git config --file=.gitmodules submodule.sub1.url "${root}/child2"
    git submodule sync -- sub1
    git submodule update --remote -- sub1
    git add sub1 .gitmodules
    git commit -m 'switch submodule url'
    popd

    #now user will update
    pushd user2
    printHeader "second user updates"
    #this will give an error, but shouldn't
    git fetch

    #test again
    git branch -r -D origin/exper
    git fetch

    #test workaround
    printHeader "workaround"
    git branch -r -D origin/exper
    git config submodule.recurse false
    git fetch

    popd
}

main $@

• What did you expect to occur after running these commands?

fetch should complete with no errors. Git for Linux 1.9 doesn't exhibit this behavior

• What actually happened instead?

fetch gave a fatal error like this

fatal: git upload-pack: not our ref <SHA>

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

NA

[dscho]

I simplified the reproducer a bit, to not require Bash:

#!/bin/sh

createRepo()
{
	git init "$1" &&
	git -C "$1" commit --allow-empty -m first
}

rm -rf child1 child2 user1 user2 &&

#create submodules
createRepo child1 &&
createRepo child2 &&

#workaround for identical initial SHA
git -C child2 commit --allow-empty -m second &&

#create main repo
createRepo user1 &&

#add submodule
git -C user1 submodule add -b master "../child1" sub1 &&
git -C user1 commit -m 'added submodule' &&

#another user clones repo
git clone --recurse-submodules user1 user2 &&

#switch submodule url in main repo in arbitrary branch
git -C user1 checkout -b exper &&
git config --file=user1/.gitmodules submodule.sub1.url "../child2" &&
git -C user1 submodule sync -- sub1 &&
git -C user1 submodule update --remote -- sub1 &&
git -C user1 commit -m 'switch submodule url' sub1 &&

#now user will update
#this will give an error, but shouldn't
git -C user2 fetch &&

#test again
git -C user2 branch -r -D origin/exper &&
git -C user2 fetch &&

#test workaround
git -C user2 branch -r -D origin/exper &&
git -C user2 config submodule.recurse false &&
git -C user2 fetch

And I can confirm that this fails in the indicated spot.

I can also confirm that this fails in just the same way on Linux, and therefore this is not a Windows-specific bug. So I looked at the Git mailing list, trying to figure out whether this has been reported already. I find a couple of hits when looking for "not our ref": https://lore.kernel.org/git/?q=%22not+our+ref%22

The most likely candidate is https://lore.kernel.org/git/5a70d535-47b0-a4ea-b4e4-572a1bcfe997@gmail.com/, but it seems that the reported use case changes the URL in the same branch. And it seems the reporter was okay with the resolution.

However, I think that the issue here is that the URL was changed on a separate branch, and that Git incorrectly thought that it had the commits in said branch available. Therefore, this looks like an upstream bug to me (my hunch is that the ref advertisement has been made a bit overzealous recently). Hence I would recommend to take it to the Git mailing list (send plain-text messages, HTML messages are dropped silently).

Your best chance will be to provide the reproducer in the form of a patch to Git's own test suite. I spent a bit more time on this than I had planned, but I think this will get you somewhere:

diff --git a/t/t3050-subprojects-fetch.sh b/t/t3050-subprojects-fetch.sh
index f1f09abdd9b..5739687a842 100755
--- a/t/t3050-subprojects-fetch.sh
+++ b/t/t3050-subprojects-fetch.sh
@@ -49,4 +49,23 @@ test_expect_success fetch '
 	test_cmp expected actual
 '
 
+test_expect_success 'with unfetched commit from new URL in other branch' '
+	test_create_repo super &&
+	git -C super submodule add ../sub &&
+	git -C super commit -m "add sub" &&
+
+	git clone --recurse-submodules super cloned-before-move &&
+
+	git clone sub new-sub &&
+	test_commit -C new-sub after-move &&
+
+	git -C super switch -c change-url &&
+	git -C super config -f .gitmodules submodule.sub.url "$PWD/new-sub" &&
+	git -C super submodule sync -- sub &&
+	git -C super submodule update --remote -- sub &&
+	git -C super commit -m "changed submodule URL" .gitmodules sub &&
+
+	git -C cloned-before-move fetch --recurse-submodules
+'
+
 test_done

[PetSerAl]

The issue as I understand it:

• Git only use submodule's own config when it need to find out submodule's fetch URL
• Git does not update fetch URL in submodule when you fetch new commits in superproject
• You do need to explicitly synchronize submodule's fetch URL with superproject

createRepo()
{
    git init "$1" &&
    git -C "$1" commit --allow-empty -m first
}

rm -rf child1 child2 user1 user2 &&

#create submodules
createRepo child1 &&
createRepo child2 &&

#workaround for identical initial SHA
git -C child2 commit --allow-empty -m second &&

#create main repo
createRepo user1 &&

#add submodule
git -C user1 submodule add -b master "../child1" sub1 &&
git -C user1 commit -m 'added submodule' &&

#another user clones repo
git clone --recurse-submodules user1 user2 &&

#switch submodule url in main repo in arbitrary branch
git -C user1 checkout -b exper &&
git config --file=user1/.gitmodules submodule.sub1.url "../child2" &&
git -C user1 submodule sync -- sub1 &&
git -C user1 submodule update --remote -- sub1 &&
git -C user1 commit -m 'switch submodule url' sub1 .gitmodules

#now user will update
git -C user2 fetch #error

git -C user2/sub1 config remote.origin.url #child1
git -C user2 checkout origin/exper
git -C user2 submodule sync
git -C user2/sub1 config remote.origin.url #child2

git -C user2 fetch --recurse-submodules #ok

[martinv-github]

@dscho Thanks for that. Guess I'll open an issue with Git as you suggest.