Skip to content

/ go

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: "fatal error: runtime: pipe failed" on macOS Big Sur #44941

Open
VojtechVitek opened this issue Mar 11, 2021 · 8 comments
Open

runtime: "fatal error: runtime: pipe failed" on macOS Big Sur #44941

VojtechVitek opened this issue Mar 11, 2021 · 8 comments
Labels
NeedsInvestigation
Milestone
Backlog

Comments

@VojtechVitek
Copy link

@VojtechVitek VojtechVitek commented Mar 11, 2021 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.16.1 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Macbook Air M1 (Apple Silicon ARM64)
macOS Big Sur go1.16.1 darwin/amd64

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/vojtechvitek/Library/Caches/go-build"
GOENV="/Users/vojtechvitek/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/vojtechvitek/go/pkg/mod"
GONOPROXY="[redacted]"
GONOSUMDB="[redacted]"
GOOS="darwin"
GOPATH="/Users/vojtechvitek/go"
GOPRIVATE="[redacted]"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/vojtechvitek/VojtechVitek/rerun/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/23/vc4gzw8n52gc17yx4wfqfj3c0000gn/T/go-build4066643553=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

$ go get github.com/VojtechVitek/rerun/cmd/rerun@v0.0.3

$ rerun -watch . -run echo "Rerunning command."
$ echo Rerunning command.
Rerunning command.
runtime: pipe failed with 91746476
fatal error: runtime: pipe failed

goroutine 6 [running]:
runtime.throw(0x10f41e8, 0x14)
	/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc00003ed38 sp=0xc00003ed08 pc=0x1034032
runtime.netpollinit()
	/usr/local/go/src/runtime/netpoll_kqueue.go:34 +0x1a5 fp=0xc00003edc0 sp=0xc00003ed38 pc=0x102fe05
runtime.netpollGenericInit()
	/usr/local/go/src/runtime/netpoll.go:123 +0x67 fp=0xc00003edd8 sp=0xc00003edc0 pc=0x102f827
runtime.doaddtimer(0xc000024000, 0xc000060058)
	/usr/local/go/src/runtime/time.go:281 +0x185 fp=0xc00003ee30 sp=0xc00003edd8 pc=0x1054905
runtime.addtimer(0xc000060058)
	/usr/local/go/src/runtime/time.go:269 +0xad fp=0xc00003ee68 sp=0xc00003ee30 pc=0x10546ed
time.startTimer(0xc000060058)
	/usr/local/go/src/runtime/time.go:212 +0x2b fp=0xc00003ee80 sp=0xc00003ee68 pc=0x1065b0b
time.NewTimer(0x7fffffffffffffff, 0x0)
	/usr/local/go/src/time/sleep.go:96 +0xc5 fp=0xc00003eec0 sp=0xc00003ee80 pc=0x109bcc5
github.com/VojtechVitek/rerun.(*Watcher).Watch.func2(0xc0000b8000, 0xbebc200, 0xc000283b60)
	/Users/vojtechvitek/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/watcher.go:88 +0x235 fp=0xc00003efc8 sp=0xc00003eec0 pc=0x10c3fb5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00003efd0 sp=0xc00003efc8 pc=0x1068841
created by github.com/VojtechVitek/rerun.(*Watcher).Watch
	/Users/vojtechvitek/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/watcher.go:82 +0x12d

goroutine 1 [chan receive]:
main.main()
	/Users/vojtechvitek/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/cmd/rerun/main.go:112 +0x3cb

goroutine 18 [syscall]:
syscall.syscall6(0x10c03c0, 0x3, 0x0, 0x0, 0xc000038688, 0xa, 0x11d0920, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/sys_darwin.go:41 +0x2e
golang.org/x/sys/unix.kevent(0x3, 0x0, 0x0, 0xc000038688, 0xa, 0x11d0920, 0x0, 0x0, 0x0)
	/Users/vojtechvitek/go/pkg/mod/golang.org/x/sys@v0.0.0-20210309074719-68d13333faf2/unix/zsyscall_darwin_amd64.go:276 +0xa5
golang.org/x/sys/unix.Kevent(0x3, 0x0, 0x0, 0x0, 0xc000038688, 0xa, 0xa, 0x11d0920, 0x0, 0x0, ...)
	/Users/vojtechvitek/go/pkg/mod/golang.org/x/sys@v0.0.0-20210309074719-68d13333faf2/unix/syscall_bsd.go:429 +0x71
github.com/fsnotify/fsnotify.read(0x3, 0xc000038688, 0xa, 0xa, 0x11d0920, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/vojtechvitek/go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:511 +0x6e
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0000b2180)
	/Users/vojtechvitek/go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:274 +0x81b
created by github.com/fsnotify/fsnotify.NewWatcher
	/Users/vojtechvitek/go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:62 +0x199

goroutine 33 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:165 +0x9d
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:151 +0x45

goroutine 5 [chan receive]:
main.main.func1(0xc0000b21e0, 0xc000056020)
	/Users/vojtechvitek/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/cmd/rerun/main.go:80 +0x3b
created by main.main
	/Users/vojtechvitek/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/cmd/rerun/main.go:79 +0x2f3
ERROR 2

What did you expect to see?

rerun -watch . -run echo "Rerunning command."
On any file-system change, the echo command would run. This works well on other architectures. And I'd swear this used to work on Go 1.16rc1. However I can't find it anymore in the Go Download page, so I can't re-test it.

What did you see instead?

fatal error: runtime: pipe failed

I'm sorry if this is insufficient bug report, but I don't have a clue what else to include. I'm not sure how to help debug this any further. Any help/guidance would be greatly appreciated.

During the crash, the program does this in the high-level:

  • Runs an os/exec command in the background
  • Listens on signals (signal.Notify)
  • Watches FS changes via kqueue (https://github.com/fsnotify/fsnotify)
  • Spins up some internal timers (time.NewTimer())

https://github.com/VojtechVitek/rerun
@mdlayher mdlayher changed the title "fatal error: runtime: pipe failed" on macOS Big Sur go1.16.1 darwin/amd64 runtime: "fatal error: runtime: pipe failed" on macOS Big Sur Mar 11, 2021
@toothrot
Copy link
Contributor

@toothrot toothrot commented Mar 11, 2021

@VojtechVitek If you'd like to verify Go 1.16RC1, you can still install it via go get:

go get golang.org/dl/go1.16rc1
go1.16rc1 download
go1.16rc1 build/install/run...

/cc @ianlancetaylor @cherrymui
@toothrot toothrot added this to the Backlog milestone Mar 11, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Mar 11, 2021

Macbook Air M1 (Apple Silicon ARM64)
macOS Big Sur go1.16.1 darwin/amd64

Is this reproducible if you run natively on ARM64 (using a toolchain built with GOARCH=arm64)? Thanks.
@VojtechVitek
Copy link
Author

@VojtechVitek VojtechVitek commented Mar 12, 2021

@cherrymui I can reproduce this on native arm64 build too (thanks for pointing this out!):

$ go version
go version go1.16.2 darwin/arm64
@VojtechVitek
Copy link
Author

@VojtechVitek VojtechVitek commented Mar 12, 2021

@toothrot I can reproduce this on Go 1.16rc1 too.

$ go1.16rc1 version
go version go1.16rc1 darwin/arm64

runtime: pipe failed with 1803448492
fatal error: runtime: pipe failed

goroutine 50 [running]:
runtime.throw(0x104a9f134, 0x14)
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/panic.go:1117 +0x54 fp=0x1400005ed00 sp=0x1400005ecd0 pc=0x104a183c4
runtime.netpollinit()
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/netpoll_kqueue.go:34 +0x190 fp=0x1400005ed90 sp=0x1400005ed00 pc=0x104a147d0
runtime.netpollGenericInit()
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/netpoll.go:123 +0x7c fp=0x1400005edb0 sp=0x1400005ed90 pc=0x104a1408c
runtime.doaddtimer(0x1400002b800, 0x1400027e008)
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/time.go:281 +0x1b8 fp=0x1400005ee10 sp=0x1400005edb0 pc=0x104a37df8
runtime.addtimer(0x1400027e008)
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/time.go:269 +0x94 fp=0x1400005ee50 sp=0x1400005ee10 pc=0x104a37bb4
time.startTimer(0x1400027e008)
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/time.go:212 +0x28 fp=0x1400005ee70 sp=0x1400005ee50 pc=0x104a47988
time.NewTimer(0x7fffffffffffffff, 0x0)
	/Users/vojtechvitek/sdk/go1.16rc1/src/time/sleep.go:96 +0xb8 fp=0x1400005eeb0 sp=0x1400005ee70 pc=0x104a78028
github.com/VojtechVitek/rerun.(*Watcher).Watch.func2(0x14000132000, 0xbebc200, 0x14000245920)
	/Users/vojtechvitek/VojtechVitek/rerun/watcher.go:88 +0x18c fp=0x1400005efc0 sp=0x1400005eeb0 pc=0x104a9b3dc
runtime.goexit()
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/asm_arm64.s:1130 +0x4 fp=0x1400005efc0 sp=0x1400005efc0 pc=0x104a4a594
created by github.com/VojtechVitek/rerun.(*Watcher).Watch
	/Users/vojtechvitek/VojtechVitek/rerun/watcher.go:82 +0xec

goroutine 1 [chan receive]:
main.main()
	/Users/vojtechvitek/VojtechVitek/rerun/cmd/rerun/main.go:112 +0x2d8

goroutine 18 [syscall]:
syscall.syscall6(0x104a97d10, 0x3, 0x0, 0x0, 0x14000038e88, 0xa, 0x104bad930, 0x0, 0x0, 0x0)
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/sys_darwin.go:41 +0x1c
golang.org/x/sys/unix.kevent(0x3, 0x0, 0x0, 0x14000038e88, 0xa, 0x104bad930, 0x0, 0x0, 0x0)
	/Users/vojtechvitek/VojtechVitek/rerun/vendor/golang.org/x/sys/unix/zsyscall_darwin_arm64.go:276 +0x80
golang.org/x/sys/unix.Kevent(0x3, 0x0, 0x0, 0x0, 0x14000038e88, 0xa, 0xa, 0x104bad930, 0x1400010e180, 0x0, ...)
	/Users/vojtechvitek/VojtechVitek/rerun/vendor/golang.org/x/sys/unix/syscall_bsd.go:429 +0x60
github.com/fsnotify/fsnotify.read(0x3, 0x14000038e88, 0xa, 0xa, 0x104bad930, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/vojtechvitek/VojtechVitek/rerun/vendor/github.com/fsnotify/fsnotify/kqueue.go:511 +0x50
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0x1400012c120)
	/Users/vojtechvitek/VojtechVitek/rerun/vendor/github.com/fsnotify/fsnotify/kqueue.go:274 +0x76c
created by github.com/fsnotify/fsnotify.NewWatcher
	/Users/vojtechvitek/VojtechVitek/rerun/vendor/github.com/fsnotify/fsnotify/kqueue.go:62 +0x15c

goroutine 33 [syscall]:
os/signal.signal_recv(0x0)
	/Users/vojtechvitek/sdk/go1.16rc1/src/runtime/sigqueue.go:165 +0xc4
os/signal.loop()
	/Users/vojtechvitek/sdk/go1.16rc1/src/os/signal/signal_unix.go:23 +0x20
created by os/signal.Notify.func1.1
	/Users/vojtechvitek/sdk/go1.16rc1/src/os/signal/signal.go:151 +0x44

goroutine 49 [chan receive]:
main.main.func1(0x1400012c180, 0x1400018c000)
	/Users/vojtechvitek/VojtechVitek/rerun/cmd/rerun/main.go:80 +0x2c
created by main.main
	/Users/vojtechvitek/VojtechVitek/rerun/cmd/rerun/main.go:79 +0x238

But I'm pretty sure I didn't run into this few weeks back.. Perhaps I might have updated Xcode Command Line Tools since then.
@cuonglm
Copy link
Contributor

@cuonglm cuonglm commented Mar 12, 2021

I got different error on an Apple M1

> rerun -watch . -run echo "Rerunning command."
$ echo Rerunning command.
Rerunning command.
runtime: kqueue failed with 1
fatal error: runtime: netpollinit failed

goroutine 35 [running]:
runtime.throw(0x10221c67b, 0x1b)
	/Users/cuonglm/sdk/go1.16.2/src/runtime/panic.go:1117 +0x54 fp=0x14000096d00 sp=0x14000096cd0 pc=0x1021943c4
runtime.netpollinit()
	/Users/cuonglm/sdk/go1.16.2/src/runtime/netpoll_kqueue.go:28 +0x1e4 fp=0x14000096d90 sp=0x14000096d00 pc=0x102190824
runtime.netpollGenericInit()
	/Users/cuonglm/sdk/go1.16.2/src/runtime/netpoll.go:123 +0x7c fp=0x14000096db0 sp=0x14000096d90 pc=0x10219008c
runtime.doaddtimer(0x14000029000, 0x14000208008)
	/Users/cuonglm/sdk/go1.16.2/src/runtime/time.go:281 +0x1b8 fp=0x14000096e10 sp=0x14000096db0 pc=0x1021b3df8
runtime.addtimer(0x14000208008)
	/Users/cuonglm/sdk/go1.16.2/src/runtime/time.go:269 +0x94 fp=0x14000096e50 sp=0x14000096e10 pc=0x1021b3bb4
time.startTimer(0x14000208008)
	/Users/cuonglm/sdk/go1.16.2/src/runtime/time.go:212 +0x28 fp=0x14000096e70 sp=0x14000096e50 pc=0x1021c3988
time.NewTimer(0x7fffffffffffffff, 0x0)
	/Users/cuonglm/sdk/go1.16.2/src/time/sleep.go:96 +0xb8 fp=0x14000096eb0 sp=0x14000096e70 pc=0x1021f4028
github.com/VojtechVitek/rerun.(*Watcher).Watch.func2(0x14000070000, 0xbebc200, 0x1400013acc0)
	/Users/cuonglm/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/watcher.go:88 +0x18c fp=0x14000096fc0 sp=0x14000096eb0 pc=0x1022173dc
runtime.goexit()
	/Users/cuonglm/sdk/go1.16.2/src/runtime/asm_arm64.s:1130 +0x4 fp=0x14000096fc0 sp=0x14000096fc0 pc=0x1021c6594
created by github.com/VojtechVitek/rerun.(*Watcher).Watch
	/Users/cuonglm/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/watcher.go:82 +0xec

goroutine 1 [chan receive]:
main.main()
	/Users/cuonglm/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/cmd/rerun/main.go:112 +0x2d8

goroutine 4 [syscall]:
syscall.syscall6(0x102213d10, 0x3, 0x0, 0x0, 0x1400003de88, 0xa, 0x102329930, 0x0, 0x0, 0x0)
	/Users/cuonglm/sdk/go1.16.2/src/runtime/sys_darwin.go:41 +0x1c
golang.org/x/sys/unix.kevent(0x3, 0x0, 0x0, 0x1400003de88, 0xa, 0x102329930, 0x0, 0x0, 0x0)
	/Users/cuonglm/go/pkg/mod/golang.org/x/sys@v0.0.0-20210309074719-68d13333faf2/unix/zsyscall_darwin_arm64.go:276 +0x80
golang.org/x/sys/unix.Kevent(0x3, 0x0, 0x0, 0x0, 0x1400003de88, 0xa, 0xa, 0x102329930, 0x1400005e180, 0x0, ...)
	/Users/cuonglm/go/pkg/mod/golang.org/x/sys@v0.0.0-20210309074719-68d13333faf2/unix/syscall_bsd.go:429 +0x60
github.com/fsnotify/fsnotify.read(0x3, 0x1400003de88, 0xa, 0xa, 0x102329930, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/cuonglm/go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:511 +0x50
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0x140000101e0)
	/Users/cuonglm/go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:274 +0x76c
created by github.com/fsnotify/fsnotify.NewWatcher
	/Users/cuonglm/go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:62 +0x15c

goroutine 33 [syscall]:
os/signal.signal_recv(0x0)
	/Users/cuonglm/sdk/go1.16.2/src/runtime/sigqueue.go:165 +0xc4
os/signal.loop()
	/Users/cuonglm/sdk/go1.16.2/src/os/signal/signal_unix.go:23 +0x20
created by os/signal.Notify.func1.1
	/Users/cuonglm/sdk/go1.16.2/src/os/signal/signal.go:151 +0x44

goroutine 34 [chan receive]:
main.main.func1(0x14000010240, 0x14000114000)
	/Users/cuonglm/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/cmd/rerun/main.go:80 +0x2c
created by main.main
	/Users/cuonglm/go/pkg/mod/github.com/!vojtech!vitek/rerun@v0.0.3/cmd/rerun/main.go:79 +0x238
@andig
Copy link
Contributor

@andig andig commented Mar 13, 2021

I've tried with with 1.16.1 on M1 arm64 and could not repro within 60s for what it's worth:

❯ uname -a
Darwin SkyNetM1.fritz.box 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 arm64

~/htdocs
❯ go get github.com/VojtechVitek/rerun/cmd/rerun@v0.0.3
Alias tip: gog github.com/VojtechVitek/rerun/cmd/rerun@v0.0.3
go: downloading github.com/VojtechVitek/rerun v0.0.3
go: downloading golang.org/x/sys v0.0.0-20210309074719-68d13333faf2

~/htdocs 6s
❯ rerun -watch . -run echo "Rerunning command."
$ echo Rerunning command.
Rerunning command.
@VojtechVitek
Copy link
Author

@VojtechVitek VojtechVitek commented Mar 16, 2021

@andig Thanks. Can you try again in a folder with many subdirectories and files? Perhaps this has something to do with the number of fsnotify (kqueue) files being watched.

I was able to run rerun command successfully without any issues in a much smaller project today (hundreds of files/subdirs rather than thousands).

@toothrot @cherrymui Is there anything else we can do or provide to help debug this issue?
@andig
Copy link
Contributor

@andig andig commented Mar 16, 2021

I've retried with 900k files (my home folder with lots of git repos inside). No crash.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation
Projects
None yet
Milestone
Backlog
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@toothrot @VojtechVitek @andig @cuonglm @cherrymui