x/vgo: issue with Git LFS

[mewmew]

What version of Go are you using (go version)?

u@x220 ~> go version
go version go1.10.2 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

[u@x220 flac]$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/u/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/foo"
GORACE=""
GOROOT="/home/u/go"
GOTMPDIR=""
GOTOOLDIR="/home/u/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build322510014=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Reproducible example for mewkiz/flac@fa00a54

[u@x220 ~]$ export GOPATH=/tmp/foo
[u@x220 ~]$ mkdir -p $GOPATH/src/github.com/mewkiz
[u@x220 ~]$ cd $GOPATH/src/github.com/mewkiz
[u@x220 mewkiz]$ git clone https://github.com/mewkiz/flac
Cloning into 'flac'...
remote: Counting objects: 1456, done.
remote: Compressing objects: 100% (79/79), done.
remote: Total 1456 (delta 59), reused 86 (delta 41), pack-reused 1332
Receiving objects: 100% (1456/1456), 5.94 MiB | 5.25 MiB/s, done.
Resolving deltas: 100% (762/762), done.
[u@x220 mewkiz]$ cd flac
[u@x220 flac]$ vgo build ./...
vgo: creating new go.mod: module github.com/mewkiz/flac
vgo: resolving import "github.com/icza/bitio"
vgo: finding github.com/icza/bitio (latest)
vgo: adding github.com/icza/bitio v0.0.0-20180221120200-b25b30b42508
vgo: resolving import "github.com/mewkiz/pkg/errutil"
vgo: finding github.com/mewkiz/pkg (latest)
vgo: adding github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: resolving import "github.com/mikkyang/id3-go/encodedbytes"
vgo: finding github.com/mikkyang/id3-go (latest)
vgo: adding github.com/mikkyang/id3-go v0.0.0-20151201011346-0168d962f1d7
vgo: resolving import "azul3d.org/engine/audio"
vgo: finding azul3d.org/engine (latest)
vgo: adding azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: finding azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: finding github.com/mikkyang/id3-go v0.0.0-20151201011346-0168d962f1d7
vgo: finding github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: finding github.com/icza/bitio v0.0.0-20180221120200-b25b30b42508
vgo: downloading github.com/icza/bitio v0.0.0-20180221120200-b25b30b42508
vgo: downloading github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: downloading github.com/mikkyang/id3-go v0.0.0-20151201011346-0168d962f1d7
vgo: downloading azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: import "github.com/mewkiz/flac/cmd/flac2wav" ->
	import "azul3d.org/engine/audio": git archive --format=zip --prefix=prefix/ eb48fbeb6d97ddebb823269e5cf41ae0af903765 in /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c: exit status 128:
	Downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (78.33 KB)
	Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e)
	
	Errors logged to /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects/logs/20180528T100042.494280683.log
	Use `git lfs logs last` to view the log.
	error: external filter 'git-lfs filter-process' failed
	fatal: audio/flac/testdata/tune_stereo_44100hz_int16.flac: smudge filter lfs failed
vgo: downloading azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: import "github.com/mewkiz/flac/cmd/flac2wav" ->
	import "azul3d.org/engine/audio/wav": git archive --format=zip --prefix=prefix/ eb48fbeb6d97ddebb823269e5cf41ae0af903765 in /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c: exit status 128:
	Downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (78.33 KB)
	Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e)
	
	Errors logged to /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects/logs/20180528T100043.050840881.log
	Use `git lfs logs last` to view the log.
	error: external filter 'git-lfs filter-process' failed
	fatal: audio/flac/testdata/tune_stereo_44100hz_int16.flac: smudge filter lfs failed
vgo: resolving import "github.com/djimenez/iconv-go"
vgo: finding github.com/djimenez/iconv-go (latest)
vgo: adding github.com/djimenez/iconv-go v0.0.0-20160305225143-8960e66bd3da

What did you expect to see?

Working build.

What did you see instead?

vgo: import "github.com/mewkiz/flac/cmd/flac2wav" ->
	import "azul3d.org/engine/audio/wav": git archive --format=zip --prefix=prefix/ eb48fbeb6d97ddebb823269e5cf41ae0af903765 in /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c: exit status 128:
	Downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (78.33 KB)
	Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e)
	
	Errors logged to /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects/logs/20180528T100043.050840881.log
	Use `git lfs logs last` to view the log.
	error: external filter 'git-lfs filter-process' failed
	fatal: audio/flac/testdata/tune_stereo_44100hz_int16.flac: smudge filter lfs failed

Log files:

[u@x220 flac]$ cat /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects/logs/20180528T100042.494280683.log
git-lfs/2.0.0 (GitHub; linux amd64; go devel +438a757d73 Wed Feb 21 18:10:00 2018 +0000)
git version 2.17.0

$ git-lfs filter-process
Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e)

Smudge error: Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e): batch response: Post /objects/batch: unsupported protocol scheme ""
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:166
github.com/git-lfs/git-lfs/errors.NewSmudgeError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:252
github.com/git-lfs/git-lfs/lfs.PointerSmudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:68
github.com/git-lfs/git-lfs/lfs.(*Pointer).Smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/pointer.go:64
github.com/git-lfs/git-lfs/commands.smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:63
github.com/git-lfs/git-lfs/commands.filterCommand
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:65
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/run.go:68
main.main
	/home/u/goget/src/github.com/github/git-lfs/git-lfs.go:35
runtime.main
	/home/u/go/src/runtime/proc.go:198
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361

ENV:
LocalWorkingDir=
LocalGitDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalGitStorageDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalMediaDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects
LocalReferenceDir=
TempDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.
GIT_PREFIX=
GIT_IMPLICIT_WORK_TREE=0
[u@x220 flac]$ cat /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects/logs/20180528T100043.050840881.log
git-lfs/2.0.0 (GitHub; linux amd64; go devel +438a757d73 Wed Feb 21 18:10:00 2018 +0000)
git version 2.17.0

$ git-lfs filter-process
Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e)

Smudge error: Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e): batch response: Post /objects/batch: unsupported protocol scheme ""
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:166
github.com/git-lfs/git-lfs/errors.NewSmudgeError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:252
github.com/git-lfs/git-lfs/lfs.PointerSmudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:68
github.com/git-lfs/git-lfs/lfs.(*Pointer).Smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/pointer.go:64
github.com/git-lfs/git-lfs/commands.smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:63
github.com/git-lfs/git-lfs/commands.filterCommand
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:65
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/run.go:68
main.main
	/home/u/goget/src/github.com/github/git-lfs/git-lfs.go:35
runtime.main
	/home/u/go/src/runtime/proc.go:198
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361

ENV:
LocalWorkingDir=
LocalGitDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalGitStorageDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalMediaDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects
LocalReferenceDir=
TempDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.
GIT_PREFIX=
GIT_IMPLICIT_WORK_TREE=0

[ttaylorr]

Hi @mewmew. I am a maintainer of Git LFS, and have seen this bug before. I am out of the office this week, but I’m the meantime can you try with the latest version of Git LFS (v2.4.2 at the time of writing)? Thanks.

[mewmew]

Hi @ttaylorr. I tried with the latest version of Git LFS, and seem to get the same error. Thanks for looking at this.

Cheers,
/u

[u@x220 flac]$ git lfs
git-lfs/2.4.0 (GitHub; linux amd64; go 1.10.2)
Sorry, no usage text found for "git-lfs"

[u@x220 ~]$ export GOPATH=/tmp/foo
[u@x220 ~]$ mkdir -p $GOPATH/src/github.com/mewkiz
[u@x220 ~]$ cd $GOPATH/src/github.com/mewkiz
[u@x220 mewkiz]$ git clone https://github.com/mewkiz/flac
Cloning into 'flac'...
remote: Counting objects: 1468, done.
remote: Compressing objects: 100% (83/83), done.
 flaceiving objects:  27% (403/1468), 1.90 MiB | 931.00 KiB/s   
remote: Total 1468 (delta 67), reused 98 (delta 48), pack-reused 1332
Receiving objects: 100% (1468/1468), 5.95 MiB | 1015.00 KiB/s, done.
Resolving deltas: 100% (770/770), done.
[u@x220 mewkiz]$ cd flac
[u@x220 flac]$ vgo build ./...
vgo: creating new go.mod: module github.com/mewkiz/flac
vgo: resolving import "github.com/icza/bitio"
vgo: finding github.com/icza/bitio (latest)
vgo: adding github.com/icza/bitio v0.0.0-20180221120200-b25b30b42508
vgo: resolving import "github.com/mewkiz/pkg/errutil"
vgo: finding github.com/mewkiz/pkg (latest)
vgo: adding github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: resolving import "github.com/mikkyang/id3-go/encodedbytes"
vgo: finding github.com/mikkyang/id3-go (latest)
vgo: adding github.com/mikkyang/id3-go v0.0.0-20151201011346-0168d962f1d7
vgo: resolving import "azul3d.org/engine/audio"
vgo: finding azul3d.org/engine (latest)
vgo: adding azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: finding azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: finding github.com/mikkyang/id3-go v0.0.0-20151201011346-0168d962f1d7
vgo: finding github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: finding github.com/icza/bitio v0.0.0-20180221120200-b25b30b42508
vgo: downloading github.com/icza/bitio v0.0.0-20180221120200-b25b30b42508
vgo: downloading github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: downloading github.com/mikkyang/id3-go v0.0.0-20151201011346-0168d962f1d7
vgo: downloading azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: import "github.com/mewkiz/flac/cmd/flac2wav" ->
	import "azul3d.org/engine/audio": git archive --format=zip --prefix=prefix/ eb48fbeb6d97ddebb823269e5cf41ae0af903765 in /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c: exit status 128:
	Downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (80 KB)
	Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f01): Smudge error: Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e): batch request: missing protocol: ""
	
	Errors logged to /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/logs/20180531T090525.388884092.log
	Use `git lfs logs last` to view the log.
	error: external filter 'git-lfs filter-process' failed
	fatal: audio/flac/testdata/tune_stereo_44100hz_int16.flac: smudge filter lfs failed
vgo: downloading azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: import "github.com/mewkiz/flac/cmd/flac2wav" ->
	import "azul3d.org/engine/audio/wav": git archive --format=zip --prefix=prefix/ eb48fbeb6d97ddebb823269e5cf41ae0af903765 in /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c: exit status 128:
	Downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (80 KB)
	Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f01): Smudge error: Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e): batch request: missing protocol: ""
	
	Errors logged to /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/logs/20180531T090525.986758053.log
	Use `git lfs logs last` to view the log.
	error: external filter 'git-lfs filter-process' failed
	fatal: audio/flac/testdata/tune_stereo_44100hz_int16.flac: smudge filter lfs failed
vgo: resolving import "github.com/djimenez/iconv-go"
vgo: finding github.com/djimenez/iconv-go (latest)
vgo: adding github.com/djimenez/iconv-go v0.0.0-20160305225143-8960e66bd3da

Edit: Contents of /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/logs/20180531T090525.388884092.log:

git-lfs/2.4.0 (GitHub; linux amd64; go 1.10.2)
git version 2.17.1

$ git-lfs filter-process
Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f01): Smudge error: Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e): batch request: missing protocol: ""

missing protocol: ""
batch request
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.Wrap
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/errors.go:74
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/api.go:62
github.com/git-lfs/git-lfs/tq.Batch
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/api.go:39
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:432
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:354
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361
Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e)
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.Wrapf
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.(*GitFilter).downloadFile
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:98
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:59
github.com/git-lfs/git-lfs/commands.smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:118
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/run.go:77
main.main
	/home/u/goget/src/github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
	/home/u/go/src/runtime/proc.go:198
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.NewSmudgeError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:256
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:68
github.com/git-lfs/git-lfs/commands.smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:118
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/run.go:77
main.main
	/home/u/goget/src/github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
	/home/u/go/src/runtime/proc.go:198
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361

Current time in UTC: 
2018-05-31 07:05:25

ENV:
LocalWorkingDir=
LocalGitDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalGitStorageDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalMediaDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects
LocalReferenceDir=
TempDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.
GIT_PREFIX=
GIT_IMPLICIT_WORK_TREE=0

Client IP addresses:
192.168.1.2 fe80::8267:56fe:5d48:19b5

Contents of /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/logs/20180531T090525.986758053.log

git-lfs/2.4.0 (GitHub; linux amd64; go 1.10.2)
git version 2.17.1

$ git-lfs filter-process
Error downloading object: audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f01): Smudge error: Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e): batch request: missing protocol: ""

missing protocol: ""
batch request
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.Wrap
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/errors.go:74
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/api.go:62
github.com/git-lfs/git-lfs/tq.Batch
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/api.go:39
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:432
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	/home/u/goget/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:354
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361
Error downloading audio/flac/testdata/tune_stereo_44100hz_int16.flac (2091f011ecd9a3af0a56a6f5a14ae7e5ffb95158b8316b7cb534dbe43b9b152e)
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.Wrapf
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.(*GitFilter).downloadFile
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:98
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:59
github.com/git-lfs/git-lfs/commands.smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:118
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/run.go:77
main.main
	/home/u/goget/src/github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
	/home/u/go/src/runtime/proc.go:198
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.NewSmudgeError
	/home/u/goget/src/github.com/git-lfs/git-lfs/errors/types.go:256
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:68
github.com/git-lfs/git-lfs/commands.smudge
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:118
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
	/home/u/goget/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/home/u/goget/src/github.com/git-lfs/git-lfs/commands/run.go:77
main.main
	/home/u/goget/src/github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
	/home/u/go/src/runtime/proc.go:198
runtime.goexit
	/home/u/go/src/runtime/asm_amd64.s:2361

Current time in UTC: 
2018-05-31 07:05:25

ENV:
LocalWorkingDir=
LocalGitDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalGitStorageDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
LocalMediaDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/objects
LocalReferenceDir=
TempDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.
GIT_PREFIX=
GIT_IMPLICIT_WORK_TREE=0

Client IP addresses:
192.168.1.2 fe80::8267:56fe:5d48:19b5

[rsc]

Hi @mewmew, @taylorr,

Without git-lfs installed on my machine, everything is working fine. I imagine that's expected, and that vgo is somehow using git-lfs in an unexpected way. I am going to send out a change to vgo to make -x show all git commands it runs, but I can tell you right now that these are the commands it runs in that repo:

mkdir -p /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
cd /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
git init --bare
git ls-remote -q https://github.com/azul3d/engine
git log -n1 --format=format:%H eb48fbeb6d97
git fetch https://github.com/azul3d/engine
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:go.mod
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:Gopkg.lock
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:GLOCKFILE
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:Godeps/Godeps.json
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:dependencies.tsv
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:glide.lock
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:vendor.conf
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:vendor.yml
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:vendor/manifest
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git cat-file blob eb48fbeb6d97ddebb823269e5cf41ae0af903765:vendor/vendor.json
git ls-remote -q https://github.com/azul3d/engine
git log -n1 --format=format:%H eb48fbeb6d97
git log -n1 --format=format:%ct eb48fbeb6d97ddebb823269e5cf41ae0af903765
git archive --format=zip --prefix=prefix/ eb48fbeb6d97ddebb823269e5cf41ae0af903765

It's that last git archive that fails, of course. It seems unlikely that the log and cat-file commands matter, but that's all of them. I wonder if the problem has to do with being a bare repo, and on top of that one not created by clone?

@mewmew, maybe it's worth trying those commands by hand, outside of vgo?

[ttaylorr]

@rsc I very much appreciate the -x patch, it made it much easier to figure out what is going on. From what I can gather, vgo fetches non-local references by repeatedly calling git-fetch(1) with more and more broad refspecs. Since git fetch does not checkout new references into the working copy (and since the repository created by vgo is --bare), objects stored in Git LFS are not cached locally.

In this case, when git-archive(1) is called later, it must first filter content from the objects that it has. When a Git LFS object is filtered, Git LFS will then proceed to attempt to download it, but fail because the repository contains no remotes and thus Git LFS is unsure of where the object came from.

I think that this can be resolved by teaching vgo to first add a remote for any non-local repository it's interested in, and then use that remote instead of a URL. In practice:

--- vgo	2018-06-06 12:27:32.000000000 -0700
+++ vgo-remote	2018-06-06 12:28:02.000000000 -0700
@@ -1,6 +1,7 @@
 mkdir -p /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
 cd /tmp/foo/src/v/cache/vcswork/33d7fd8670cc26328c70a1aa2e256f39e8677b5051ce504802f38deaa8cce53c
 git init --bare
+git remote add origin https://github.com/azul3d/engine
-git ls-remote -q https://github.com/azul3d/engine
+git ls-remote -q origin
 git log -n1 --format=format:%H eb48fbeb6d97
-git fetch https://github.com/azul3d/engine
+git fetch origin

If this is an approach that you think would be OK for vgo, I would be happy to write a patch to that effect.

[gopherbot]

Change https://golang.org/cl/116756 mentions this issue: cmd/go/internal/modfetch/codehost: show git commands with -x

[gopherbot]

Change https://golang.org/cl/116758 mentions this issue: cmd/go/internal/modfetch/gitrepo: fix problem with Git LFS

[rsc]

Sure, we can adjust the command being run to do that. I'll take care of it. Change sent. Thanks.

[ttaylorr]

@rsc wonderful, CL 116758 works great for me:

# (in vgo)
$ git show --oneline -q HEAD
426db6d (HEAD -> master) cmd/go/internal/modfetch/gitrepo: fix problem with Git LFS

$ git clone https://github.com/mewkiz/flac
# ...

# (in mewkiz/flac.git)
$ vgo build ./...
vgo: creating new go.mod: module github.com/mewkiz/flac
vgo: resolving import "github.com/icza/bitio"
vgo: finding github.com/icza/bitio (latest)
vgo: adding github.com/icza/bitio v0.0.0-20180221120200-b25b30b42508
vgo: resolving import "github.com/mewkiz/pkg/errutil"
vgo: finding github.com/mewkiz/pkg (latest)
vgo: adding github.com/mewkiz/pkg v0.0.0-20170428133157-c19a6f9fa6f8
vgo: resolving import "github.com/mikkyang/id3-go/encodedbytes"
vgo: finding github.com/mikkyang/id3-go (latest)
vgo: adding github.com/mikkyang/id3-go v0.0.0-20151201011346-0168d962f1d7
vgo: resolving import "azul3d.org/engine/audio"
vgo: finding azul3d.org/engine (latest)
vgo: adding azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: downloading azul3d.org/engine v0.0.0-20170701203912-eb48fbeb6d97
vgo: resolving import "github.com/djimenez/iconv-go"
vgo: finding github.com/djimenez/iconv-go (latest)
vgo: adding github.com/djimenez/iconv-go v0.0.0-20160305225143-8960e66bd3da
vgo: finding github.com/djimenez/iconv-go v0.0.0-20160305225143-8960e66bd3da
vgo: downloading github.com/djimenez/iconv-go v0.0.0-20160305225143-8960e66bd3da

Thanks! 🙇

[ttaylorr]

For posterity: I think that any repositories being built by vgo depend on a Git LFS object during compilation will fail, since those objects will not be checked out, (because the repository is bare). That's a limitation of git-archive(1) and Git LFS, but not vgo.