cmd/link: issues with Xcode 16 beta [freeze exception]

[cherrymui]

Xcode 16 beta is released recently. I downloaded it and tested against Go (running all.bash). There are a number of failures. Filing this issue to track.

On ARM64, when external linking, it seems to cause code sign error, where binaries from go build have invalid code signature and get killed.

% go build ~/go/c.go
% codesign -v ./c
./c: invalid signature (code or signature have been modified)
In architecture: arm64

Besides, there are a few failures below.

A number of failures seems to be DWARF related

--- FAIL: TestDWARF (0.00s)
    --- FAIL: TestDWARF/testprogcgo (3.89s)
        dwarf_test.go:155: decoding dwarf section info at offset 0x0: too short
FAIL
FAIL	cmd/link	27.419s

--- FAIL: TestRuntimeTypeAttrExternal (9.12s)
    dwarf_test.go:110: ## build output:
        # command-line-arguments
        ld: warning: no platform load command found in '/private/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/go-link-3081649599/go.o', assuming: macOS
    dwarf_test.go:942: error reading DWARF: decoding dwarf section info at offset 0x0: too short
FAIL
FAIL	cmd/link/internal/ld	37.212s

--- FAIL: TestDWARFSections (0.40s)
    plugin_test.go:210: /Users/cherryyz/src/go/bin/go run -gcflags  ./checkdwarf/main.go plugin2.so plugin2.UnexportedNameReuse: exit status 1
        plugin2.so: error opening DWARF: decoding dwarf section info at offset 0x0: too short
        exit status 1
FAIL
FAIL	cmd/cgo/internal/testplugin	70.410s

And there is also one that seems not DWARF related

--- FAIL: TestExtLinkCmdlineDeterminism (12.92s)
    link_test.go:1216: build failed: exit status 1, output:
        # command-line-arguments
        build mode: pie, symbol table: on, DWARF: on
        HEADER = -H1 -T0x1001000 -R0x1000
        /Users/cherryyz/src/go/pkg/tool/darwin_amd64/link: cannot create /var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/TestExtLinkCmdlineDeterminism2689331464/001/linktmp/go.o: open /var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/TestExtLinkCmdlineDeterminism2689331464/001/linktmp/go.o: no such file or directory

They are failing with ld-1115.5.3. I haven't investigated them and I'm not sure they are bugs in Go or Apple's linker. I'll look into them.

[gabyhelp]

Similar Issues

• cmd/link: TestDWARF failing under all.bash on OSX 10.13.6 #30236
• cmd/link: TestDWARF fails with the latest tip version #25392
• cmd/link: issues with Apple's new linker in Xcode 15 beta #61229
• cmd/link: no room to add dwarf info #41967
• cmd/link: testDWARF in c-archive mode fails on Linux (but not Darwin) #31363
• cmd/link: revisit dwarf in executable files on OS X #11994
• cmd/link: vmsize of __DWARF should not be zero on darwin #21647
• plugin: test fails for debug sections on Darwin #27502
• debug/dwarf: cgo produces malformed DWARF data #32673
• cmd/link: Darwin build fails with "unknown load command" #25908

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[gopherbot]

Change https://go.dev/cl/593659 mentions this issue: cmd/link: don't let dsymutil delete our temp directory

[gopherbot]

Change https://go.dev/cl/593660 mentions this issue: cmd/link: don't skip code sign even if dsymutil didn't generate a file

[cherrymui]

To work around #59026, where dsymutil may not clean up its temp directory at exit, we set DSYMUTIL_REPRODUCER_PATH to our temp directory. The problem is that in Xcode 16 beta, dsymutil deletes the DSYMUTIL_REPRODUCER_PATH directory even if it is not empty. In particular, that's where our DWARF file (the output of dsymutil) locates. That causes all the DWARF missing failures.

When the linker is invoked with -tmpdir set explicitly, that also causes the temp dir being deleted (by dsymutil), which causes the TestExtLinkCmdlineDeterminism failure.

CL 593659 fixes this, by giving dsymutil a subdirectory. This makes all.bash pass with Xcode 16 beta on both Intel and ARM64.

The code sign failure is also related. As dsymutil deletes the directory, which contains its own output DWARF file, which makes the Go linker not be able to find the file and exit too early. With the fix, this should not happen. But even if this happens (dsymutil didn't generate an output), we should not skip code sign. CL 593660 addresses this.

[cherrymui]

@golang/release I'm requesting a freeze exception for this. Xcode 16 beta doesn't work well with Go currently, with the most significant problem being the code sign error on ARM64, which makes almost all cgo binaries fail to run. This is caused by dsymutil unexpectedly (but arguable) deleting the temp directory (details above). Xcode 16 is still in beta, and Apple might change this before the final release, or not. Xcode 16 will probably be released later this year, well before Go 1.24. Instead of waiting for Apple to fix, we can do a simple workaround in Go 1.23. This ensures Go 1.23 works well when Xcode 16 is released, and also make users who want to use Apple's beta toolchain happy. The fix is simple and low risk, just changing the temp directory to pass to dsymutil. Thanks.

(This doesn't need to block RC.)

[cherrymui]

@gopherbot please backport this to Go 1.22. Thanks.

Reason is similar to the freeze exception above. Xcode 16 will probably be released while Go 1.22 is still in support.

[gopherbot]

Backport issue(s) opened: #68198 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[joedian]

@cherrymui approved.