Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: gopls -v crashes immediately when linked with Xcode 15 beta #61190

Closed
mattmassicotte opened this issue Jul 6, 2023 · 33 comments
Closed
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@mattmassicotte
Copy link

gopls version

Tested both 0.11.0 and 0.12.4. Cannot capture version output, as it crashes immediately on launch. Output is:

fatal: morestack on g0

go env

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/matt/Library/Caches/go-build"
GOENV="/Users/matt/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/matt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/matt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.20.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9_/q1rgv_js2rb518ql_svng32c0000gn/T/go-build3080324972=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Built gopls 0.12.4 with 1.20.5. Ran "gopls -v". Crashes immediately. Things were working fine on beta 2. Now, normally, I'd just wait because this is a beta. But, I always get worried when stuff like this happens, so I figured I'd at least make you aware.

@gopherbot gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Jul 6, 2023
@gopherbot gopherbot added this to the Unreleased milestone Jul 6, 2023
@findleyr
Copy link
Contributor

findleyr commented Jul 6, 2023

Hi, just to be sure I understand, these are the steps you followed:

  1. install macOS 14 beta 3 (https://developer.apple.com/news/releases/?id=07052023c)
  2. build gopls@v0.12.4 (latest), using the pre-compiled mac binaries for go 1.20.5
  3. run gopls -v: crash

Is that right? Is there any output before the crash?

@mattmassicotte
Copy link
Author

Those are the steps I followed. But, I had first tried a version of gopls I had previously built with an earlier Go version/macOS and that also wasn't able to launch.

All combinations produced the same output:

fatal: morestack on g0

@findleyr
Copy link
Contributor

findleyr commented Jul 6, 2023

Thanks again (and sorry I missed that you'd already included the output).

CC @golang/runtime for awareness. I'm not sure if this is something we'd normally escalate to Apple.

Are you experiencing this type of problem with other Go binaries?

@findleyr findleyr changed the title x/tools/gopls: gopls crashes on macOS 14 beta 3 runtime: gopls -v crashes immediately on macOS 14 beta 3 Jul 6, 2023
@findleyr findleyr added compiler/runtime Issues related to the Go compiler and/or runtime. gopls Issues related to the Go language server, gopls. and removed gopls Issues related to the Go language server, gopls. labels Jul 6, 2023
@mattmassicotte
Copy link
Author

So far, I have not seen this occur with any other go binaries, but my testing has been pretty limited. I've also filed a bug with Apple about it.

@prattmic
Copy link
Member

prattmic commented Jul 6, 2023

Would it be possible for you to clone https://github.com/golang/go, checkout tag go1.20.5 and then run src/all.bash to run all of the Go tests to see if any of those fail?

@mattmassicotte
Copy link
Author

Just ran them, and in fact, they fail.

Building Go cmd/dist using /opt/homebrew/Cellar/go/1.20.5/libexec. (go1.20.5 darwin/arm64)
Building Go toolchain1 using /opt/homebrew/Cellar/go/1.20.5/libexec.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/arm64.

##### Test execution environment.
# GOARCH: arm64
# CPU: 
# GOOS: darwin
# OS Version: Darwin 23.0.0 Darwin Kernel Version 23.0.0: Fri Jun 30 17:50:12 PDT 2023; root:xnu-10002.0.168.505.3~1/RELEASE_ARM64_T6000 arm64

##### Testing packages.
ok  	archive/tar	0.189s
ok  	archive/zip	0.299s
ok  	bufio	0.143s
ok  	bytes	0.371s
ok  	compress/bzip2	0.291s
ok  	compress/flate	0.706s
ok  	compress/gzip	1.549s
ok  	compress/lzw	0.068s
ok  	compress/zlib	0.503s
ok  	container/heap	0.240s
ok  	container/list	0.337s
ok  	container/ring	0.289s
ok  	context	0.198s
ok  	crypto	0.385s
ok  	crypto/aes	0.445s
ok  	crypto/cipher	0.253s
ok  	crypto/des	0.258s
ok  	crypto/dsa	0.310s
ok  	crypto/ecdh	0.300s
ok  	crypto/ecdsa	0.087s
ok  	crypto/ed25519	0.203s
ok  	crypto/elliptic	0.220s
ok  	crypto/hmac	0.116s
ok  	crypto/internal/alias	0.055s
ok  	crypto/internal/bigmod	0.196s
ok  	crypto/internal/boring	0.144s
ok  	crypto/internal/boring/bcache	0.235s
ok  	crypto/internal/edwards25519	2.492s
ok  	crypto/internal/edwards25519/field	2.662s
ok  	crypto/internal/nistec	0.463s
ok  	crypto/internal/nistec/fiat	0.265s [no tests to run]
ok  	crypto/md5	0.297s
ok  	crypto/rand	0.283s
ok  	crypto/rc4	0.259s
ok  	crypto/rsa	0.432s
ok  	crypto/sha1	0.111s
ok  	crypto/sha256	0.063s
ok  	crypto/sha512	0.107s
ok  	crypto/subtle	0.126s
ok  	crypto/tls	0.330s
ok  	crypto/x509	0.330s
ok  	database/sql	0.322s
ok  	database/sql/driver	0.307s
ok  	debug/buildinfo	0.470s
ok  	debug/dwarf	0.423s
ok  	debug/elf	0.315s
ok  	debug/gosym	0.360s
ok  	debug/macho	0.333s
ok  	debug/pe	0.179s
ok  	debug/plan9obj	0.174s
ok  	embed	0.208s [no tests to run]
ok  	embed/internal/embedtest	0.252s
ok  	encoding/ascii85	0.277s
ok  	encoding/asn1	0.238s
ok  	encoding/base32	0.299s
ok  	encoding/base64	0.059s
ok  	encoding/binary	0.112s
ok  	encoding/csv	0.066s
ok  	encoding/gob	0.669s
ok  	encoding/hex	0.130s
ok  	encoding/json	0.194s
ok  	encoding/pem	0.639s
ok  	encoding/xml	0.186s
ok  	errors	0.227s
ok  	expvar	0.286s
ok  	flag	0.385s
ok  	fmt	0.114s
ok  	go/ast	0.070s
ok  	go/build	1.246s
ok  	go/build/constraint	0.252s
ok  	go/constant	0.173s
ok  	go/doc	0.338s
ok  	go/doc/comment	1.039s
ok  	go/format	0.414s
ok  	go/importer	0.590s
ok  	go/internal/gccgoimporter	0.394s
ok  	go/internal/gcimporter	1.296s
ok  	go/internal/srcimporter	7.546s
ok  	go/parser	0.271s
ok  	go/printer	0.267s
ok  	go/scanner	0.073s
ok  	go/token	0.073s
ok  	go/types	2.492s
ok  	hash	0.260s
ok  	hash/adler32	0.363s
ok  	hash/crc32	0.311s
ok  	hash/crc64	0.210s
ok  	hash/fnv	0.132s
ok  	hash/maphash	0.500s
ok  	html	0.412s
ok  	html/template	0.349s
ok  	image	0.114s
ok  	image/color	0.121s
ok  	image/draw	0.118s
ok  	image/gif	0.197s
ok  	image/jpeg	0.168s
ok  	image/png	0.209s
ok  	index/suffixarray	0.218s
ok  	internal/abi	1.070s
ok  	internal/buildcfg	0.207s
ok  	internal/coverage/cformat	0.306s
ok  	internal/coverage/cmerge	0.159s
ok  	internal/coverage/pods	0.267s
ok  	internal/coverage/slicereader	0.356s
ok  	internal/coverage/slicewriter	0.458s
ok  	internal/coverage/test	0.114s
ok  	internal/cpu	0.065s
ok  	internal/dag	0.269s
ok  	internal/diff	0.218s
ok  	internal/fmtsort	0.111s
ok  	internal/fuzz	0.168s
ok  	internal/godebug	0.373s
ok  	internal/intern	0.686s
ok  	internal/itoa	0.316s
ok  	internal/poll	0.111s
ok  	internal/profile	0.122s
ok  	internal/reflectlite	0.084s
ok  	internal/safefilepath	0.331s
ok  	internal/saferio	0.444s
ok  	internal/singleflight	0.401s
ok  	internal/testenv	0.175s
ok  	internal/trace	0.249s
ok  	internal/types/errors	0.337s
ok  	internal/unsafeheader	0.474s
ok  	internal/xcoff	0.279s
ok  	io	0.104s
ok  	io/fs	0.504s
ok  	io/ioutil	0.218s
ok  	log	0.165s
ok  	log/syslog	1.513s
ok  	math	0.072s
ok  	math/big	0.664s
ok  	math/bits	0.070s
ok  	math/cmplx	0.118s
ok  	math/rand	0.275s
ok  	mime	0.168s
ok  	mime/multipart	0.633s
ok  	mime/quotedprintable	0.306s
ok  	net	2.215s
ok  	net/http	2.578s
ok  	net/http/cgi	1.363s
ok  	net/http/cookiejar	0.250s
ok  	net/http/fcgi	0.463s
ok  	net/http/httptest	0.492s
ok  	net/http/httptrace	0.302s
ok  	net/http/httputil	1.084s
ok  	net/http/internal	0.639s
ok  	net/http/internal/ascii	0.589s
# plugin.test
ld: warning: -bind_at_load is deprecated on macOS
ld: warning: '/private/var/folders/9_/q1rgv_js2rb518ql_svng32c0000gn/T/go-link-4136702197/go.o' has malformed LC_DYSYMTAB, expected 91 undefined symbols to start at index 15225, found 97 undefined symbols starting at index 69
ok  	net/http/pprof	4.301s
ok  	net/internal/socktest	0.321s
ok  	net/mail	0.182s
ok  	net/netip	0.303s
ok  	net/rpc	0.330s
ok  	net/rpc/jsonrpc	0.153s
ok  	net/smtp	0.436s
ok  	net/textproto	0.495s
ok  	net/url	0.204s
ok  	os	0.844s
ok  	os/exec	0.328s
ok  	os/exec/internal/fdtest	0.287s
ok  	os/signal	2.894s
ok  	os/user	0.189s
ok  	path	0.335s
ok  	path/filepath	0.091s
ok  	plugin	0.091s
ok  	reflect	0.475s
ok  	regexp	0.402s
ok  	regexp/syntax	0.623s
ok  	runtime	28.409s
ok  	runtime/cgo	0.320s
ok  	runtime/coverage	0.266s
ok  	runtime/debug	0.624s
ok  	runtime/internal/atomic	0.258s
ok  	runtime/internal/math	0.363s
ok  	runtime/internal/sys	0.467s
ok  	runtime/metrics	0.411s
ok  	runtime/pprof	6.636s
ok  	runtime/trace	2.314s
ok  	sort	0.097s
ok  	strconv	0.394s
ok  	strings	0.165s
ok  	sync	0.369s
ok  	sync/atomic	0.636s
ok  	syscall	0.153s
ok  	testing	1.000s
ok  	testing/fstest	0.285s
ok  	testing/iotest	0.327s
ok  	testing/quick	0.191s
ok  	text/scanner	0.231s
ok  	text/tabwriter	0.386s
ok  	text/template	0.161s
ok  	text/template/parse	0.171s
ok  	time	2.602s
ok  	unicode	0.291s
ok  	unicode/utf16	0.127s
ok  	unicode/utf8	0.178s
ok  	cmd/addr2line	0.635s
ok  	cmd/api	5.450s
ok  	cmd/asm/internal/asm	0.557s
ok  	cmd/asm/internal/lex	0.174s
ok  	cmd/compile/internal/abt	0.210s
ok  	cmd/compile/internal/amd64	0.173s
ok  	cmd/compile/internal/base	0.218s
ok  	cmd/compile/internal/compare	0.086s
ok  	cmd/compile/internal/dwarfgen	0.518s
ok  	cmd/compile/internal/importer	1.700s
ok  	cmd/compile/internal/ir	0.248s
ok  	cmd/compile/internal/logopt	0.428s
ok  	cmd/compile/internal/noder	0.241s
ok  	cmd/compile/internal/reflectdata	0.157s [no tests to run]
ok  	cmd/compile/internal/ssa	33.958s
ok  	cmd/compile/internal/syntax	0.187s
ok  	cmd/compile/internal/test	10.216s
ok  	cmd/compile/internal/typecheck	0.762s
ok  	cmd/compile/internal/types	0.327s
ok  	cmd/compile/internal/types2	7.927s
ok  	cmd/covdata	0.579s
ok  	cmd/cover	2.490s
ok  	cmd/dist	0.443s
ok  	cmd/doc	0.511s
ok  	cmd/fix	3.800s
ok  	cmd/go	35.257s
ok  	cmd/go/internal/auth	0.094s
ok  	cmd/go/internal/cache	0.302s
ok  	cmd/go/internal/fsys	0.104s
ok  	cmd/go/internal/generate	0.139s
ok  	cmd/go/internal/get	0.249s
ok  	cmd/go/internal/imports	0.263s
ok  	cmd/go/internal/load	0.115s
ok  	cmd/go/internal/lockedfile	0.224s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.116s
ok  	cmd/go/internal/modconv	0.067s
ok  	cmd/go/internal/modfetch	0.122s
ok  	cmd/go/internal/modfetch/codehost	0.112s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.241s
ok  	cmd/go/internal/modindex	0.479s
ok  	cmd/go/internal/modload	0.125s
ok  	cmd/go/internal/mvs	0.217s
ok  	cmd/go/internal/par	0.270s
ok  	cmd/go/internal/str	0.166s
ok  	cmd/go/internal/test	0.144s
ok  	cmd/go/internal/vcs	0.128s
ok  	cmd/go/internal/vcweb	0.302s
ok  	cmd/go/internal/vcweb/vcstest	5.502s
ok  	cmd/go/internal/web	0.329s
ok  	cmd/go/internal/work	0.137s
ok  	cmd/gofmt	0.212s
ok  	cmd/internal/archive	10.283s
ok  	cmd/internal/buildid	0.458s
ok  	cmd/internal/cov	1.487s
ok  	cmd/internal/dwarf	0.135s
ok  	cmd/internal/edit	0.345s
ok  	cmd/internal/goobj	0.365s
ok  	cmd/internal/moddeps	3.465s
ok  	cmd/internal/notsha256	0.295s
ok  	cmd/internal/obj	0.794s
ok  	cmd/internal/obj/arm64	0.249s
ok  	cmd/internal/obj/ppc64	0.608s
ok  	cmd/internal/obj/riscv	0.244s
ok  	cmd/internal/obj/s390x	0.088s
ok  	cmd/internal/obj/x86	5.406s
ok  	cmd/internal/objabi	0.282s
ok  	cmd/internal/pkgpath	0.231s
ok  	cmd/internal/pkgpattern	0.227s
ok  	cmd/internal/quoted	0.351s
ok  	cmd/internal/src	0.387s
ok  	cmd/internal/test2json	0.468s
--- FAIL: TestDWARF (0.00s)
    --- FAIL: TestDWARF/testprogcgo (1.89s)
        dwarf_test.go:170: ErrUnknownPC
FAIL
FAIL	cmd/link	13.695s
ok  	cmd/link/internal/benchmark	0.071s
--- FAIL: TestRuntimeTypeAttrExternal (2.65s)
    dwarf_test.go:105: ## build output:
        # command-line-arguments
        ld: warning: '/private/var/folders/9_/q1rgv_js2rb518ql_svng32c0000gn/T/go-link-644603662/go.o' has malformed LC_DYSYMTAB, expected 44 undefined symbols to start at index 1548, found 50 undefined symbols starting at index 18
    dwarf_test.go:806: wanted 1 DIE named *main.X, found 0
FAIL
FAIL	cmd/link/internal/ld	13.489s
ok  	cmd/link/internal/loader	0.315s
ok  	cmd/nm	2.239s
ok  	cmd/objdump	7.029s
ok  	cmd/pack	3.898s
ok  	cmd/pprof	0.187s
ok  	cmd/trace	0.676s
ok  	cmd/vet	6.458s
FAIL
go tool dist: Failed: exit status 1

@cherrymui
Copy link
Member

Most of the errors in #61190 (comment) should be addressed with CL stack https://golang.org/cl/505415 . I think they are caused by the (buggy or intentional or a mixture of) behavior changes of Apple's new linker. What version of the C toolchain and Xcode are you using? Could you share the output of ld -v? (I thought the new Apple linker has not been made default, maybe it does?)

fatal: morestack on g0

This is more concerning. Does this occur only with previously-built binaries, or it also occurs with new binaries?

I'll update my macOS machine to the beta and see tonight...

@findleyr
Copy link
Contributor

findleyr commented Jul 6, 2023

Removing the gopls label to hide this from our triage dashboard. Please re-add it if this appears to be directly related to gopls in any way.

@findleyr findleyr removed the gopls Issues related to the Go language server, gopls. label Jul 6, 2023
@mattmassicotte
Copy link
Author

I am using the latest Xcode 15 beta.

$ ld -v
@(#)PROGRAM:ld-classic  PROJECT:ld64-906.2
BUILD 20:45:38 Jun 29 2023
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 15.0.0 (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 15.0.0 (tapi-1500.0.12)

I've also confirmed that an older copy of gopls I built with some version of macOS 13 + pre Xcode 15 does seem to work correctly on macOS 14 beta 3.

@cherrymui
Copy link
Member

Thanks. I found that while ld -v still shows the old linker, the C compiler automatically switches to the new one. cc -Wl,-v would print something like

% cc -Wl,-v   
@(#)PROGRAM:ld  PROJECT:dyld-1009.5
BUILD 20:45:24 Jun 29 2023
...

I filed #61229 for the linker issue. Let's keep this issue for the fatal: morestack on g0 failure and gopls doesn't run. For a failing binary, do you know what version of the Go toolchain was used to build it? (go version <binary> may be helpful.) If you know the C toolchain version that would be even better. Thanks!

@mattmassicotte
Copy link
Author

I changed a few things while trying to get it working. I'm now building with 1.20.5. What information can I provide to help understand the C toolchain? I've installed Xcode 15 beta 3. I ran your cc command, which seems slightly unhappy on my machine...

$cc -Wl,-v
@(#)PROGRAM:ld  PROJECT:dyld-1009.5
BUILD 20:45:24 Jun 29 2023
configured to support archs: i386 x86_64 x86_64h armv6 armv7 armv7s armv7m armv7k arm64 arm64e arm64_32
LTO support using: LLVM version 15.0.0 (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 15.0.0 (tapi-1500.0.12)
Library search paths:
	/usr/local/lib
Framework search paths:
ld: Undefined symbols:
  _main, referenced from:

clang: error: linker command failed with exit code 1 (use -v to see invocation)

@cherrymui
Copy link
Member

I'm now building with 1.20.5.

Does it work? Or still crash?

I ran your cc command, which seems slightly unhappy on my machine...

That command is expected to fail, no worries. @(#)PROGRAM:ld PROJECT:dyld-1009.5 is the version number.

Do you still have the binary that fails with fatal: morestack on g0? If so, what is the Go toolchain version that builds the binary (run go version <binary>)? And do you know/remember the C toolchain version that builds the binary?
Thanks.

@mattmassicotte
Copy link
Author

mattmassicotte commented Jul 10, 2023

It crashes when built with 1.20.5, in the same way. I've rebuilt in numerous times, and it always crashes. I tracked down a previous copy I built (built on different OS, toolchain, and Go version) and it runs ok.

How exactly can I get the C toolchain version that you are looking for? I just use whatever comes packaged with Xcode. Currently using Xcode 15 beta 3.

@cherrymui
Copy link
Member

Thanks. If it is a newly built binary crashing, then nothing more needed to find out the C toolchain version. It is Xcode 15 beta 3, which is sufficient.

Could you confirm that gopls is the binary that is crashing, and other binaries don't? Thanks.

@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 10, 2023
@cherrymui cherrymui modified the milestones: Unreleased, Go1.22 Jul 10, 2023
@cherrymui cherrymui removed the Tools This label describes issues relating to any tools in the x/tools repository. label Jul 10, 2023
@mattmassicotte
Copy link
Author

I haven't done a lot of testing, but I compiled a few other simple Go programs and they all worked ok.

@AverageMarcus
Copy link

Just want to point out that this isn't just an issue for macOS 14. 😞

I have macOS 13.4.1 but the latest beta version of Xcode CommandLineTools installed as it was suggested via the mac software update tool. Not sure why it's installing beta versions of xcode tools (I didn't realise until I hit this issue)

$ clang -v
Apple clang version 15.0.0 (clang-1500.0.34.3)
Target: arm64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Screenshot 2023-07-11 at 10 54 07 am

$ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: 15.0.0.0.1.1688827124
volume: /
location: /
install-time: 1689068473

If anyone else is in the same situation as me I was able to resolve it by downloading the old CommandLineTools from here: https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_for_Xcode_14.3.1/Command_Line_Tools_for_Xcode_14.3.1.dmg

@cherrymui
Copy link
Member

@AverageMarcus what is the failure you see? gopls crash, or something else? Thanks.

@AverageMarcus
Copy link

For me it was when trying to build my own codebase but with related issue.

The actual error I got was:

/opt/homebrew/Cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: warning: -bind_at_load is deprecated on macOS
ld: warning: '/private/var/folders/w0/swt9pf957fd1yb8x3tphq5640000gn/T/go-link-4006510922/go.o' has malformed LC_DYSYMTAB, expected 138 undefined symbols to start at index 178490, found 147 undefined symbols starting at index 118
0  0x100bd4380  __assert_rtn + 72
1  0x100b8d9bc  ___ZN2ld16LayoutExecutable27writeContentWithoutLinkEditENSt3__14spanIhLm18446744073709551615EEEy_block_invoke + 9620
2  0x19b3bc440  _dispatch_client_callout2 + 20
3  0x19b3cff1c  _dispatch_apply_invoke + 224
4  0x19b3bc400  _dispatch_client_callout + 20
5  0x19b3cdfb8  _dispatch_root_queue_drain + 684
6  0x19b3ce6c0  _dispatch_worker_thread2 + 164
7  0x19b568038  _pthread_wqthread + 228
ld: Assertion failed: (addr + content.size() <= sectionEndAddr), function writeContentWithoutLinkEdit_block_invoke, file Layout.cpp, line 5689.
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The problem seems to have been introduced (as far as I can tell) in the latest xcode Command Line Tools beta.

@cherrymui
Copy link
Member

@AverageMarcus thanks. See issue #61229. The CLs linked there should include a workaround. You're welcome to try that and let us know if it still fails. Thanks.

@AverageMarcus
Copy link

Oh nice! I didn't see that one. Thanks.

@cherrymui
Copy link
Member

I'm able to reproduce to gopls crash, linking with the new Apple's linker ld-prime (even running on macOS 13). It appears CLs linked at #61229 also fix this (I haven't identified which CL matters most). Thanks.

@cherrymui cherrymui self-assigned this Jul 12, 2023
@cherrymui
Copy link
Member

The morestack on g0 failure is from an infinite recursion

^C* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
  * frame #0: 0x0000000100075102 gopls`go:buildid + 457090
    frame #1: 0x0000000100073365 gopls`runtime.morestack.abi0 + 37
    frame #2: 0x000000010003cb69 gopls`runtime.panicCheck1 + 169
    frame #3: 0x000000010005d705 gopls`runtime.findfunc + 261
    frame #4: 0x000000010003cae5 gopls`runtime.panicCheck1 + 37
    frame #5: 0x000000010003cc0e gopls`runtime.goPanicIndex + 46
    frame #6: 0x000000010005d705 gopls`runtime.findfunc + 261
    frame #7: 0x000000010003cae5 gopls`runtime.panicCheck1 + 37
    frame #8: 0x000000010003cc0e gopls`runtime.goPanicIndex + 46
    frame #9: 0x000000010005d705 gopls`runtime.findfunc + 261

which looks like runtime func table is probably corrupted...

@cherrymui cherrymui changed the title runtime: gopls -v crashes immediately on macOS 14 beta 3 runtime: gopls -v crashes immediately when linked with Xcode 15 beta Jul 24, 2023
@cherrymui
Copy link
Member

cherrymui commented Jul 24, 2023

I found that some relocations are resolved to wrong addresses by the new system linker ld-prime. E.g. a field in runtime.firstmoduledata is expected to point to runtime.pclntab+0x334620 (which is what we have in the object file passed to Apple's linker), but in the executable it points to runtime.pclntab+0xa33c0. I haven't been able to identify a pattern.

This causes the runtime func table to be incorrect, which causes the runtime to fail to verify the table and fall into infinite recursion at start.

CLs linked at #61229 changes to use a different form of relocation, which is probably what "fixes" the issue.

@wtfiscrq
Copy link

@cherrymui I’m currently facing this issue in my machine. If you can provide steps to test your changes I can give it a shot. 🙏 Thanks!

@bobdoah
Copy link

bobdoah commented Jul 25, 2023

Thanks @AverageMarcus, downgrading worked for me. I couldn't see a work around described in any of the linked CLs.

@cherrymui
Copy link
Member

cherrymui commented Jul 25, 2023

@wtfiscrq issue #61229 linked a number of CLs. If you want to test you can checkout them and build a new Go toolchain. One way of doing it is git fetch https://go.googlesource.com/go refs/changes/55/511355/2 && git checkout FETCH_HEAD then run make.bash in GOROOT/src.

@bobdoah the CLs include workarounds in the Go toolchain. You need to checkout the CLs and build a new Go toolchain (as mentioned above). If you don't want to build the Go toolchain yourself, here are some other workarounds:
go build -ldflags="-extldflags=-Wl,-ld64" - force using the old Apple linker (update: in Xcode 15 beta 5 the flag is switched to -ld_classic, so the workaround is go build -ldflags="-extldflags=-Wl,-ld_classic")
go build -ldflags=-linkmode=internal - force internal linking, don't use Apple linker at all

@cherrymui
Copy link
Member

I submitted a few CLs to the master branch. Now it seems to work fine with Go tip on Intel Mac. Note that you'll need to build PIE binary with go build -buildmode=pie, as non-PIE build is broken in the new Apple linker.

It still fails on ARM64 Mac. Some CLs linked at #61229 will work around it, but not yet submitted. One way to checkout the CLs are

% go install golang.org/dl/gotip@latest
% gotip download 511355

The run gotip as the go command, such as gotip build.

@annguyen17-tiki
Copy link

annguyen17-tiki commented Aug 3, 2023

The same problem happens with me when running gopls from VS Code. Here is what I have worked around while waiting for an official fix.

Delete the crashed version of gopls from GOPATH (the <GOPATH>/bin/gopls file), then install another gopls package using homebrew

brew install gopls

You can check whether it is installed successfully by using the command gopls version (restart your terminal if needed)

Finally, copy gopls executable file from your brew path (default is /opt/homebrew/bin/gopls, or you can run the command which gopls to check that info) to your GOPATH, which will be later read by VS Code.

cp <HOMEBREW_PATH>/bin/gopls <GOPATH>/bin/gopls

Restart VS Code and done.

@shreyasbhat0
Copy link

Thanks @annguyen17-tiki ,this worked for me :)

@qiquanlu
Copy link

qiquanlu commented Aug 9, 2023

@cherrymui
I have two M1 MBP, both on most recent Sonoma Beta, with go version go1.21.0 darwin/arm64. I'm having trouble with one of the machine when running gopls version crashes showing fatal: morestack on g0. however on the other machine, everything seems fine.

I'm willing to check log/versions on my laptops if this would help on tracing the issue.

@domino14
Copy link

domino14 commented Aug 9, 2023

@annguyen17-tiki This did not work for me. Every time I open VS Code after copying the executable from homebrew, it does:

Tools environment: GOPATH=/Users/me/go
Installing 1 tool at /Users/me/go/bin in module mode.
  gopls

I've already copied it there, why does it always try to replace it? and it always replaces it with a broken version.

@domino14
Copy link

domino14 commented Aug 9, 2023

I also can't downgrade the XCode tools version. The download page didn't work, and wants me to pay $99 to be an Apple Developer??????

@cherrymui
Copy link
Member

I think the bug is fixed in Xcode 15 beta 6. With beta 6, gopls builds and runs fine with Go tip, 1.21.0, and 1.20.7, both on ARM64 and AMD64. So I think we are done with this. If one still sees a failure, feel free to comment. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Development

No branches or pull requests