Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

git lfs crash on macOS Sierra 10.12 (Build 16A286a) #1440

Closed
daveanderson opened this issue Aug 12, 2016 · 15 comments
Closed

git lfs crash on macOS Sierra 10.12 (Build 16A286a) #1440

daveanderson opened this issue Aug 12, 2016 · 15 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@technoweenie @nathany @ttaylorr @daveanderson @david6p2
@daveanderson
Copy link

@daveanderson daveanderson commented Aug 12, 2016

When I attempt to git lfs pull on machine running macOS Sierra 10.12 (Build 16A286a) with git-lfs/1.3.1 (GitHub; darwin amd64; go 1.6.3; git 9c9dffb) I get the following crash.

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x21d03ec97d28 pc=0x1039b]

runtime stack:
runtime.throw(0x62e660, 0x2a)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/sigpanic_unix.go:12 +0x5a
runtime.unlock(0x836680)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/lock_sema.go:107 +0x14b
runtime.(*mheap).freeSpan.func1()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mheap.go:726 +0x186
runtime.systemstack(0x70000fea0b68)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/asm_amd64.s:307 +0xab
runtime.(*mheap).freeSpan(0x836680, 0x97aae8, 0x1)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mheap.go:727 +0x44
runtime.(*mspan).sweep(0x97aae8, 0x1a300000100, 0x1)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mgcsweep.go:342 +0x6ef
runtime.sweepone(0x0)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mgcsweep.go:112 +0x23e
runtime.gosweepone.func1()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mgcsweep.go:124 +0x21
runtime.systemstack(0x70000fea0d98)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/asm_amd64.s:307 +0xab
runtime.gosweepone(0x8376e8)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mgcsweep.go:125 +0x3d
runtime.deductSweepCredit(0x2000, 0x0)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mgcsweep.go:384 +0xc6
runtime.(*mcentral).cacheSpan(0x837b10, 0xc820026c00)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mcentral.go:36 +0x56
runtime.(*mcache).refill(0x9234b0, 0x4, 0x100000)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/mcache.go:119 +0xcc
runtime.mallocgc.func2()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/malloc.go:642 +0x2b
runtime.systemstack(0xc820018000)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/asm_amd64.s:291 +0x79
runtime.mstart()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/proc.go:1051

goroutine 22 [running]:
runtime.systemstack_switch()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/asm_amd64.s:245 fp=0xc8200325f0 sp=0xc8200325e8
runtime.mallocgc(0x30, 0x53e3c0, 0x0, 0xc820032700)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/malloc.go:643 +0x869 fp=0xc8200326c8 sp=0xc8200325f0
runtime.newobject(0x53e3c0, 0x10cd90)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/malloc.go:781 +0x42 fp=0xc8200326f0 sp=0xc8200326c8
runtime.makemap(0x417580, 0x0, 0x0, 0x0, 0xc820126800)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/hashmap.go:257 +0x381 fp=0xc820032750 sp=0xc8200326f0
github.com/github/git-lfs/errutil.newWrappedError(0x973028, 0xc8202ea040, 0x601ce0, 0x21, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/errutil/errors.go:327 +0x1f4 fp=0xc820032840 sp=0xc820032750
github.com/github/git-lfs/errutil.NewNotAPointerError(0x0, 0x0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/errutil/errors.go:518 +0x50 fp=0xc8200328a0 sp=0xc820032840
github.com/github/git-lfs/lfs.decodeKVData(0xc820027800, 0x15d, 0x400, 0xc8201be030, 0x0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/lfs/pointer.go:236 +0xd9 fp=0xc820032af8 sp=0xc8200328a0
github.com/github/git-lfs/lfs.decodeKV(0xc820027800, 0x15d, 0x400, 0xc820027800, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/lfs/pointer.go:139 +0x5f fp=0xc820032d28 sp=0xc820032af8
github.com/github/git-lfs/lfs.DecodeFrom(0x9b30a0, 0xc8200e0070, 0x0, 0x0, 0x0, 0x523c00, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/lfs/pointer.go:120 +0x17b fp=0xc820032dc0 sp=0xc820032d28
github.com/github/git-lfs/lfs.DecodePointer(0x9b30a0, 0xc8200e0070, 0xc820074580, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/lfs/pointer.go:107 +0x37 fp=0xc820032e08 sp=0xc820032dc0
github.com/github/git-lfs/lfs.catFileBatch.func1(0xc8201181a0, 0xc8200eb0b0, 0xc82011a5a0, 0xc82011a600)
    /Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:563 +0x5f0 fp=0xc820032f90 sp=0xc820032e08
runtime.goexit()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820032f98 sp=0xc820032f90
created by github.com/github/git-lfs/lfs.catFileBatch
    /Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:585 +0x1be

goroutine 1 [chan receive]:
github.com/github/git-lfs/lfs.ScanRefs(0xc82005ed70, 0x28, 0x0, 0x0, 0xc8200eae70, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:114 +0x164
github.com/github/git-lfs/commands.pointersToFetchForRef(0xc82005ed70, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/commands/command_fetch.go:127 +0x13a
github.com/github/git-lfs/commands.fetchRefToChan(0xc82005ed70, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/commands/command_fetch.go:132 +0x60
github.com/github/git-lfs/commands.pull(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/commands/command_pull.go:47 +0xe0
github.com/github/git-lfs/commands.pullCommand(0x82cdc0, 0x84e578, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/commands/command_pull.go:36 +0x2a6
github.com/github/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute(0x82cdc0, 0x84e578, 0x0, 0x0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/vendor/github.com/spf13/cobra/command.go:477 +0x3f4
github.com/github/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute(0x82e2e0, 0x0, 0x0)
    /Users/rick/go/src/github.com/github/git-lfs/vendor/github.com/spf13/cobra/command.go:551 +0x46a
github.com/github/git-lfs/commands.Run()
    /Users/rick/go/src/github.com/github/git-lfs/commands/commands.go:112 +0x23
main.main()
    /Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:33 +0x12e

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/Cellar/go/1.6.3/libexec/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/Cellar/go/1.6.3/libexec/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0x674f58, 0xc820022f28, 0x58bad0, 0x6, 0xfe118, 0x2)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820022f28, 0x0, 0x18)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820022f28)
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/signal1_unix.go:279 +0x32c
runtime.goexit()
    /usr/local/Cellar/go/1.6.3/libexec/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
main.main.func1(0xc8200178c0, 0xc82000fb10)
    /Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:21 +0x66
created by main.main
    /Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:31 +0x129

goroutine 24 [chan receive]:
github.com/github/git-lfs/lfs.ScanRefsToChan.func2(0xc820118280, 0xc8200eae70, 0xc82011a660, 0xc82011a6c0)
    /Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:157 +0x58
created by github.com/github/git-lfs/lfs.ScanRefsToChan
    /Users/rick/go/src/github.com/github/git-lfs/lfs/scanner.go:169 +0x357

I'm not in a position to interpret this stack trace and would appreciate any recommendations for having this work on Sierra.
@ttaylorr
Copy link
Contributor

@ttaylorr ttaylorr commented Aug 12, 2016

Hi @daveanderson -- thanks for opening this. Go 1.6.x has had a number of problems on macOS Sierra. I know that there were some fixes targeted at 1.7.0 that were backported into 1.6.3, but it looks like there are still problems.

I'm thinking that this has to do with the underlying channel-range mechanism provided by Go, since two of the goroutines in the dump are halted on range operators over channels, and there are other mentions of low-level standard library stuff.

I don't have macOS Sierra running on my computer, but if you're interested in compiling Git LFS under Go 1.7rc6 and seeing what happens, that would be 👍

@daveanderson
Copy link
Author

@daveanderson daveanderson commented Aug 12, 2016

@ttaylorr I did recompile Git LFS under Go 1.7rc6 and I get

Fetching MyBranchName
Git LFS: (0 of 1 files) 0 B / 119.08 MB
exit status 255
exit status 255

and no log file is created. It's possible that due to environment issues I'm trying to fetch a branch for which this account doesn't have permissions to access the repo, but with no log file created and just a 255 exit code I'm at a loss (and copied the file over manually...)

@ttaylorr
Copy link
Contributor

@ttaylorr ttaylorr commented Aug 12, 2016

Hmm, this is very strange behavior. Are you able to reproduce the problem on El Capitan? My hunch is that this is a macOS Sierra problem, and that the best thing we can do is report compatibility issues back to the Go team.

@nathany
Copy link

@nathany nathany commented Aug 12, 2016

Hi Dave. Is build 16A286a the beta 4 of Sierra or beta 5?

There were a number of issues caused by Go's support for gettimeofday. Go 1.6.3 addressed issues seen in beta 3 of Sierra and Go 1.7 rc6 addressed new issues seen in beta 4 of Sierra.

There are currently no open Sierra issues in the Go 1.7 milestone which is due out on Monday if no new issues are reported. https://github.com/golang/go/milestone/31

Also no Sierra related fixes since Go 1.7 rc6 was released.
golang/go@release-branch.go1.7...master

So if indeed there is a issue with Go 1.7, it's important that we get it reported right away if we want to see the fix in 1.7 -- otherwise it may end up waiting for the release of Sierra and a future Go patch release (outside of installing Go from source).

I'm currently running El Capitan 10.11.6. Let me know if I can be of assistance tracking this issue down.

@daveanderson
Copy link
Author

@daveanderson daveanderson commented Aug 12, 2016

The issue is with Sierra Beta 5. I'm using Xcode8b5 and Xcode Server b3 (the latest). This is a pretty complicated setup so I'll see ASAP if I can reduce the setup to something basic and reproducible.

@ttaylorr
Copy link
Contributor

@ttaylorr ttaylorr commented Aug 12, 2016

@nathany Thanks for hopping in. @technoweenie and I are setting up a fresh installation of macOS Sierra on a new computer now. We are going to attempt to reproduce this issue both on Sierra and El Capitan, and determine whether or not it's a Go issue, or a LFS one.

More soon.

@ttaylorr
Copy link
Contributor

@ttaylorr ttaylorr commented Aug 12, 2016

@daveanderson Once you've set up a test case, can you send instructions over on how to reproduce it so we can test side-by-side?

@nathany
Copy link

@nathany nathany commented Aug 12, 2016

Not sure if this helps, but it seems the exit code 255 would be coming from:
https://github.com/github/git-lfs/blob/master/git-lfs.go#L29

However, that would imply a signal of 127 or 0x7F, which I don't see here:
https://golang.org/pkg/syscall/#SIGABRT

Maybe that error code is from somewhere else?

It will be much easier to track this down once we have a test case. Thanks for jumping on this @ttaylorr and @technoweenie.

@nathany
Copy link

@nathany nathany commented Aug 12, 2016 •
edited

127 "command not found"

Though the initial crash was definitely due to a bug in Go (that will require a Go 1.7 build once released), this later issue may just be configuration -- like not finding git credential or something? (Likely two things)

@daveanderson
Copy link
Author

@daveanderson daveanderson commented Aug 12, 2016 •
edited

ok, I've created a sample repo that includes a 119 MB file that uses git-lfs. Doing a clone of this repo on a VM running Sierra 16A286a using a build of Git LFS using Go 1.7rc6 works fine.

$git clone https://github.com/daveanderson/git-lfs-go-sierra.git
Cloning into 'git-lfs-go-sierra'...
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 9 (delta 1), reused 8 (delta 0), pack-reused 0
Unpacking objects: 100% (9/9), done.
Checking connectivity... done.
Downloading a.log (119.00 MB)

$ git-lfs env
git-lfs/1.3.1 (GitHub; darwin amd64; go 1.7rc6; git 525d07e)
git version 2.8.4 (Apple Git-73)

Endpoint=https://github.com/daveanderson/git-lfs-go-sierra.git/info/lfs (auth=none)
...

The original issue appears to be a go 1.6.3 issue that seems to be resolved with go 1.7rc6.
I suspect that secondary issue is related to a permissions issue. It is unfortunate that additional logging does not exist to make the exit code of 255 clear in this situation, however this is not scope of the original issue.

Setting up a test case that replicates the second issue is non-trivial and I may be able to resolve the secondary issue through an alternative means.

@technoweenie
Copy link
Contributor

@technoweenie technoweenie commented Aug 12, 2016

We've confirmed that git-lfs's integration tests pass on go 1.7r6 and Sierra beta 5. It doesn't seem like there's a Go 1.7 and macOS Sierra issue that needs to hold up a Go release.

@daveanderson: based on your stack trace, I wonder if it's coming from lfs/scanner.go:585. The scanner calls git cat-file to find the LFS objects in a range of git commits. We can work with you next week to get a reproducible test case, and hopefully a fix. Protip: You can run GIT_TRACE=1 git lfs ... to get more internal debug messages from git and lfs.

@nathany Thanks for chiming in! Our next LFS release will likely be on go 1.7.

@nathany
Copy link

@nathany nathany commented Aug 15, 2016

Go 1.7 is out.
https://blog.golang.org/go1.7

@technoweenie
Copy link
Contributor

@technoweenie technoweenie commented Aug 16, 2016

Closing this. @daveanderson if you're still having issues, please post more details about what you're doing. Include the full command that's running Git LFS with GIT_TRACE=1 enabled.

@ttaylorr ttaylorr mentioned this issue Sep 11, 2016
Closed
@david6p2
Copy link

@david6p2 david6p2 commented Apr 27, 2017 •
edited

Having the same error with

$ git-lfs env:
    git-lfs/1.3.1 (GitHub; darwin amd64; go 1.6.3; git 9c9dffb)
    git version 2.10.1

I will try to update go

@ttaylorr
Copy link
Contributor

@ttaylorr ttaylorr commented Apr 27, 2017

@david6p2 the LFS version that you're running is a few out of date. The latest is 2.0.2, and yours is built against a version of Go that is known to not work with macOS Sierra.

Please run brew update && brew upgrade git-lfs and try again. If you encounter any further issues, don't hesitate to open a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment