LFS fetches objects from wrong remote during pull

[derrickstolee]

Test Setup:
The following setup requires two local copies of the repo (repo1, repo2) and two remote copies ("origin" and "dev").

• Clone an LFS repo. The remote is "origin". Clone into both repo1 and repo2.
• In repo1, change an LFS-tracked file and commit.
• Create a new remote "dev" that points to a different location.
• Push to "dev" in repo1.
• Pull from "dev" in repo2.

Expected Result:
LFS uses the "dev" URL to create the batch URL.

Actual Result:
LFS uses the "origin" URL to create the batch URL.

Version: 2.0.1.

[derrickstolee]

The situation gets exacerbated if I git remote remove origin as LFS chooses its URL as "/objects/batch":

> POST /objects/batch HTTP/1.1
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 125
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.0.1 (GitHub; windows amd64; go 1.8; git 678cdbd4)
>
trace git-lfs: api error: Post /objects/batch: unsupported protocol scheme ""

[ttaylorr]

Hi @derrickstolee, thanks for opening up this issue and sorry that you're having trouble. I have a few questions that I'd like to look at before diving in:

• Push to "dev" in repo1.
• Pull from "dev" in repo2.

With these commands, were these git push/pull commands or git lfs push/pull commands? Could you attach a log of all of these command runs using the GIT_TRACE=1 environment flag, and post the output here?

I tried my hand at reproducing this issue myself with the following test-case, but was unable to reproduce the problem:

#!/usr/bin/env bash

. "test/testlib.sh"

begin_test "push two remotes"
(
  set -e

  reponame="multiple-remotes"
  setup_remote_repo "$reponame"
  setup_remote_repo "$reponame-etc"

  clone_repo "$reponame" "$reponame"
  git remote add etc "$GITSERVER/$reponame-etc"

  git lfs track "*.dat"
  git add .gitattributes
  git commit -m "initial commit"

  contents="contents"
  contents_oid="$(calc_oid "$contents")"

  printf "$contents" > a.dat
  git add a.dat
  git commit -m "add a.dat"

  git lfs push etc master

  refute_server_object "$reponame" "$contents_oid"
  assert_server_object "$reponame-etc" "$contents_oid"
)
end_test

[derrickstolee]

@ttaylorr sorry that my repo steps weren't very clear. I don't fully understand your test infrastructure, so I hope you will accept the following repro steps as a Bash script (feel free to exchange the two remotes with your own copies):

#!/bin/bash

mkdir Issue2105-1

cd Issue2105-1
git init
git lfs track "*.txt"
echo blah > file.txt
git add file.txt .gitattributes
git commit -m "Initial commit"

git remote add origin https://stolee.visualstudio.com/DefaultCollection/LFS/_git/Issue2105A
git push -f origin master

cd ..
git clone https://stolee.visualstudio.com/DefaultCollection/LFS/_git/Issue2105A Issue2105-2
cd Issue2105-2
echo blerg > file2.txt
git add file2.txt
git commit -m "Second commit"
git remote add dev https://stolee.visualstudio.com/DefaultCollection/LFS/_git/Issue2105B
git push -f dev master

cd ../Issue2105-1
git remote add dev https://stolee.visualstudio.com/DefaultCollection/LFS/_git/Issue2105B
git pull dev master

I have the full logs available, but here is the important part from the git pull dev master command:

09:34:15.416932 run-command.c:1169      run_processes_parallel: preparing to run up to 1 tasks
09:34:15.416932 run-command.c:1201      run_processes_parallel: done
09:34:15.416932 run-command.c:369       trace: run_command: 'gc' '--auto'
From https://stolee.visualstudio.com/DefaultCollection/LFS/_git/Issue2105B
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> dev/master
09:34:15.483431 git.c:501               trace: built-in: git 'gc' '--auto'
09:34:15.490931 run-command.c:369       trace: run_command: 'merge' 'FETCH_HEAD'
09:34:15.546430 git.c:501               trace: built-in: git 'merge' 'FETCH_HEAD'
Updating b5c7bb4..31608cd
09:34:15.551931 run-command.c:369       trace: run_command: 'git-lfs filter-process'
trace git-lfs: run_command: 'git' version
trace git-lfs: run_command: 'git' config -l
trace git-lfs: Initialize filter-process
trace git-lfs: Read filter-process request.
Downloading file2.txt (6 B)
trace git-lfs: tq: running as batched queue, batch size of 100
trace git-lfs: tq: sending batch of size 1
trace git-lfs: api: batch 1 files
trace git-lfs: Filled credentials for https://stolee.visualstudio.com/DefaultCollection/LFS/_git/Issue2105A.git/info/lfs
trace git-lfs: HTTP: POST https://stolee.visualstudio.com/DefaultCollection/LFS/_git/Issue2105A.git/info/lfs/objects/batch
> POST /DefaultCollection/LFS/_git/Issue2105A.git/info/lfs/objects/batch HTTP/1.1
> Host: stolee.visualstudio.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 120
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.0.1 (GitHub; windows amd64; go 1.8; git 678cdbd4)
> 
{"operation":"download","objects":[{"oid":"9b4ccd8f1d28e77162f6a09e94e8abbc3b26a538bdd703146fbcfc6143d8d32e","size":6}]}trace git-lfs: HTTP: 200

Notice that the batch call goes to the Issue2105A remote and not the Issue2105B remote.

[ttaylorr]

@derrickstolee interesting. I have a few ideas of what might be going on. Could you post the output of git lfs env in the the Issue2105-1 repo?

[derrickstolee]

Here is the result of git lfs env:

git-lfs/2.0.1 (GitHub; windows amd64; go 1.8; git 678cdbd4)
git version 2.12.0.gvfs.1.9.gc7cc6e9

LocalWorkingDir=
LocalGitDir=
LocalGitStorageDir=
LocalMediaDir=
LocalReferenceDir=
TempDir=C:\Users\destole\AppData\Local\Temp\git-lfs
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_CURL_VERBOSE=1
GIT_LFS_PATH=C:\Program Files\Git LFS
GIT_TRACE=1
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

I know my Git version is weird. It should behave the same as 2.12.0 with regard to LFS.

[ttaylorr]

I think this is an interaction with GVFS, or something else. None of these lines should be empty:

LocalWorkingDir=
LocalGitDir=
LocalGitStorageDir=
LocalMediaDir=
LocalReferenceDir=

[derrickstolee]

I repro'd this on a different computer that has never had a GVFS version of Git4W. Here is the git lfs env output for that machine:

git-lfs/2.0.1 (GitHub; windows amd64; go 1.8; git 678cdbd4)
git version 2.12.2.windows.1

LocalWorkingDir=
LocalGitDir=
LocalGitStorageDir=
LocalMediaDir=
LocalReferenceDir=
TempDir=C:\Users\stole\AppData\Local\Temp\git-lfs
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_CURL_VERBOSE=1
GIT_TRACE=1
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

I see that again those lines are empty in my environment, but I'm not sure how that applies to this issue.

[ttaylorr]

@derrickstolee are you using the git lfs smudge (or git lfs filter-process) command directly? These are known to not working by themselves with multiple remotes, since neither command takes a remote argument. You could use git lfs pull <remote> [branch] instead.

[derrickstolee]

I never did anything to set up the smudges directly. I just expect git pull <remote> [branch] to work correctly.

[technoweenie]

The problem is that Git has forgotten the last remote it's used by the time it invokes the smudge filter after running git pull. So LFS has to guess, and assumes the default. This is where a Git change would be welcome. Maybe after running git pull <remote>, Git could set GIT_REMOTE=<remote> in the env when calling the smudge filter...?