Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: issues with Xcode 16 beta [freeze exception] #68088

Closed
cherrymui opened this issue Jun 20, 2024 · 11 comments
Closed

cmd/link: issues with Xcode 16 beta [freeze exception] #68088

cherrymui opened this issue Jun 20, 2024 · 11 comments
Assignees
@cherrymui
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.23

Comments

@cherrymui
Copy link
Member

cherrymui commented Jun 20, 2024 •
edited
Loading

Xcode 16 beta is released recently. I downloaded it and tested against Go (running all.bash). There are a number of failures. Filing this issue to track.

On ARM64, when external linking, it seems to cause code sign error, where binaries from go build have invalid code signature and get killed.

% go build ~/go/c.go
% codesign -v ./c
./c: invalid signature (code or signature have been modified)
In architecture: arm64

Besides, there are a few failures below.

A number of failures seems to be DWARF related

--- FAIL: TestDWARF (0.00s)
    --- FAIL: TestDWARF/testprogcgo (3.89s)
        dwarf_test.go:155: decoding dwarf section info at offset 0x0: too short
FAIL
FAIL	cmd/link	27.419s

--- FAIL: TestRuntimeTypeAttrExternal (9.12s)
    dwarf_test.go:110: ## build output:
        # command-line-arguments
        ld: warning: no platform load command found in '/private/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/go-link-3081649599/go.o', assuming: macOS
    dwarf_test.go:942: error reading DWARF: decoding dwarf section info at offset 0x0: too short
FAIL
FAIL	cmd/link/internal/ld	37.212s

--- FAIL: TestDWARFSections (0.40s)
    plugin_test.go:210: /Users/cherryyz/src/go/bin/go run -gcflags  ./checkdwarf/main.go plugin2.so plugin2.UnexportedNameReuse: exit status 1
        plugin2.so: error opening DWARF: decoding dwarf section info at offset 0x0: too short
        exit status 1
FAIL
FAIL	cmd/cgo/internal/testplugin	70.410s

And there is also one that seems not DWARF related

--- FAIL: TestExtLinkCmdlineDeterminism (12.92s)
    link_test.go:1216: build failed: exit status 1, output:
        # command-line-arguments
        build mode: pie, symbol table: on, DWARF: on
        HEADER = -H1 -T0x1001000 -R0x1000
        /Users/cherryyz/src/go/pkg/tool/darwin_amd64/link: cannot create /var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/TestExtLinkCmdlineDeterminism2689331464/001/linktmp/go.o: open /var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/TestExtLinkCmdlineDeterminism2689331464/001/linktmp/go.o: no such file or directory

They are failing with ld-1115.5.3. I haven't investigated them and I'm not sure they are bugs in Go or Apple's linker. I'll look into them.
@cherrymui cherrymui self-assigned this Jun 20, 2024
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jun 20, 2024
@gabyhelp
Copy link

Similar Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/593659 mentions this issue: cmd/link: don't let dsymutil delete our temp directory

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/593660 mentions this issue: cmd/link: don't skip code sign even if dsymutil didn't generate a file

@cherrymui
Copy link
Member Author

cherrymui commented Jun 21, 2024 •
edited by gabyhelp
Loading

To work around #59026, where dsymutil may not clean up its temp directory at exit, we set DSYMUTIL_REPRODUCER_PATH to our temp directory. The problem is that in Xcode 16 beta, dsymutil deletes the DSYMUTIL_REPRODUCER_PATH directory even if it is not empty. In particular, that's where our DWARF file (the output of dsymutil) locates. That causes all the DWARF missing failures.

When the linker is invoked with -tmpdir set explicitly, that also causes the temp dir being deleted (by dsymutil), which causes the TestExtLinkCmdlineDeterminism failure.

CL 593659 fixes this, by giving dsymutil a subdirectory. This makes all.bash pass with Xcode 16 beta on both Intel and ARM64.

The code sign failure is also related. As dsymutil deletes the directory, which contains its own output DWARF file, which makes the Go linker not be able to find the file and exit too early. With the fix, this should not happen. But even if this happens (dsymutil didn't generate an output), we should not skip code sign. CL 593660 addresses this.

@cherrymui cherrymui changed the title cmd/link: issues with Xcode 16 beta cmd/link: issues with Xcode 16 beta [freeze exception] Jun 21, 2024
@cherrymui
Copy link
Member Author

cherrymui commented Jun 21, 2024 •
edited
Loading

@golang/release I'm requesting a freeze exception for this. Xcode 16 beta doesn't work well with Go currently, with the most significant problem being the code sign error on ARM64, which makes almost all cgo binaries fail to run. This is caused by dsymutil unexpectedly (but arguable) deleting the temp directory (details above). Xcode 16 is still in beta, and Apple might change this before the final release, or not. Xcode 16 will probably be released later this year, well before Go 1.24. Instead of waiting for Apple to fix, we can do a simple workaround in Go 1.23. This ensures Go 1.23 works well when Xcode 16 is released, and also make users who want to use Apple's beta toolchain happy. The fix is simple and low risk, just changing the temp directory to pass to dsymutil. Thanks.

(This doesn't need to block RC.)

@dmitshur dmitshur added this to the Go1.23 milestone Jun 21, 2024
@cherrymui
Copy link
Member Author

@gopherbot please backport this to Go 1.22. Thanks.

Reason is similar to the freeze exception above. Xcode 16 will probably be released while Go 1.22 is still in support.

@gopherbot
Copy link
Contributor

Backport issue(s) opened: #68198 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@gopherbot gopherbot mentioned this issue Jun 26, 2024
Closed
@joedian
Copy link

joedian commented Jun 26, 2024

@cherrymui approved.

@cherrymui
Copy link
Member Author

Thanks!

gopherbot pushed a commit that referenced this issue Jun 26, 2024
@cherrymui
cmd/link: don't let dsymutil delete our temp directory
5f319b7 
To work around #59026, where dsymutil may not clean up its temp
directory at exit, we set DSYMUTIL_REPRODUCER_PATH to our temp
directory so it uses that, and we can delete it at the end.

In Xcode 16 beta, dsymutil deletes the DSYMUTIL_REPRODUCER_PATH
directory even if it is not empty. We still need our tmpdir at the
point, so give a subdirectory to dsymutil instead.

For #68088.

Change-Id: I18759cc39512819bbd0511793ce917eae72245d6
Reviewed-on: https://go-review.googlesource.com/c/go/+/593659
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
gopherbot pushed a commit that referenced this issue Jun 26, 2024
@cherrymui
cmd/link: don't skip code sign even if dsymutil didn't generate a file
5a18e79 
Even if dsymutil didn't generate a file (which should not happen
with the Apple toolchain with the correct setup), we should not
skip next steps, e.g. code sign. A return statement makes it exit
too early.

Updates #68088.

Change-Id: Ic1271ed1b7fe5bdee5a25cc5d669a105173b389e
Reviewed-on: https://go-review.googlesource.com/c/go/+/593660
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Than McIntosh <thanm@google.com>
@cherrymui
Copy link
Member Author

Fixed by the CLs above.

@dmitshur dmitshur added the NeedsFix The path to resolution is known, but the work has not been done. label Jun 26, 2024
@rsc rsc changed the title cmd/link: issues with Xcode 16 beta [freeze exception] cmd/link: issues with Xcode 16 beta [freeze exception] Jun 28, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/596455 mentions this issue: [release-branch.go1.22] cmd/link: don't let dsymutil delete our temp directory

Mchnan pushed a commit to Mchnan/go-sylixos that referenced this issue Jul 9, 2024
@cherrymui
cmd/link: don't let dsymutil delete our temp directory
760e687 
To work around golang#59026, where dsymutil may not clean up its temp
directory at exit, we set DSYMUTIL_REPRODUCER_PATH to our temp
directory so it uses that, and we can delete it at the end.

In Xcode 16 beta, dsymutil deletes the DSYMUTIL_REPRODUCER_PATH
directory even if it is not empty. We still need our tmpdir at the
point, so give a subdirectory to dsymutil instead.

For golang#68088.

Change-Id: I18759cc39512819bbd0511793ce917eae72245d6
Reviewed-on: https://go-review.googlesource.com/c/go/+/593659
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Mchnan pushed a commit to Mchnan/go-sylixos that referenced this issue Jul 9, 2024
@cherrymui
cmd/link: don't skip code sign even if dsymutil didn't generate a file
a738a8c 
Even if dsymutil didn't generate a file (which should not happen
with the Apple toolchain with the correct setup), we should not
skip next steps, e.g. code sign. A return statement makes it exit
too early.

Updates golang#68088.

Change-Id: Ic1271ed1b7fe5bdee5a25cc5d669a105173b389e
Reviewed-on: https://go-review.googlesource.com/c/go/+/593660
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Than McIntosh <thanm@google.com>
gopherbot pushed a commit that referenced this issue Jul 10, 2024
@cherrymui @cagedmantis
[release-branch.go1.22] cmd/link: don't let dsymutil delete our temp …
4e548f2 
…directory

To work around #59026, where dsymutil may not clean up its temp
directory at exit, we set DSYMUTIL_REPRODUCER_PATH to our temp
directory so it uses that, and we can delete it at the end.

In Xcode 16 beta, dsymutil deletes the DSYMUTIL_REPRODUCER_PATH
directory even if it is not empty. We still need our tmpdir at the
point, so give a subdirectory to dsymutil instead.

Updates #68088.
Fixes #68198.

Change-Id: I18759cc39512819bbd0511793ce917eae72245d6
Reviewed-on: https://go-review.googlesource.com/c/go/+/593659
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
(cherry picked from commit 5f319b7)
Reviewed-on: https://go-review.googlesource.com/c/go/+/596455
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cherrymui cherrymui

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.23
Development

No branches or pull requests
5 participants
@dmitshur @gopherbot @cherrymui @joedian @gabyhelp