Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: use libsystem_kernel.dylib or libSystem.dylib for syscalls on macOS #17490

Closed
copumpkin opened this Issue Oct 17, 2016 · 34 comments

Comments

Projects
None yet
10 participants
@copumpkin
Copy link

commented Oct 17, 2016

As I understand it, Go currently has its own syscall wrappers for Darwin. This is explicitly against what Apple recommends, precisely because they're not willing to commit to a particular syscall ABI. This leads to issues like #16570, and although we've been lucky in that things have generally been backward-compatible so far, there's no guarantee that it'll continue to happen. It doesn't seem inconceivable to me that we'd at some point end up having to specify "build for macOS 10.13+" vs. "build for 10.12 and below", for example.

Linking against libsystem_kernel.dylib (or the broader libSystem.dylib) would put Go back in line with Apple's recommendations for the platform, and the library is guaranteed to exist on all macOS boxes.

Apologies if this has been suggested before and people have already gone over good reasons not to do it, but I'm currently struggling to figure out what appears to be another Sierra incompatibility related to #16570 above (still haven't figured it out enough to post a reproducible bug) and am wishing we didn't have to deal with these issues.

@ianlancetaylor ianlancetaylor changed the title Consider linking against libsystem_kernel.dylib or libSystem.dylib on macOS? cmd/link: link against libsystem_kernel.dylib or libSystem.dylib on macOS Oct 18, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2016

I agree that we have to do this. I'm not sure who is actually going to do it, though.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2016

Note that the trouble with #16570 comes from fetching the current time, and that is something that is so performance critical that we might want to continue to use our own implementation even if we use libSystem for other syscall.

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2016

We got into a discussion about how to do this over on #17200. I believe this can be done without forcing external linking. But I also don't know who is going to do it, my plate is full for the foreseeable future.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2017

Agree we should do this at some point. It's not terribly high priority though.

@rsc rsc added NeedsFix and removed NeedsDecision labels Jan 4, 2017

@rsc rsc changed the title cmd/link: link against libsystem_kernel.dylib or libSystem.dylib on macOS cmd/link: use libsystem_kernel.dylib or libSystem.dylib for syscalls on macOS Mar 8, 2017

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2017

This is about using the system calls. We already link against libSystem.dylib when cgo is in use.

@hirochachacha

This comment has been minimized.

Copy link
Contributor

commented Jul 20, 2017

Hi, I made a CL for x86. https://go-review.googlesource.com/c/50290/
Could someone review it (after development cycle open) ?
If the overall design is OK, I'd like do something in go 1.10. Thank you.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 20, 2017

CL https://golang.org/cl/50290 mentions this issue.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9Maybe Jul 20, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Aug 11, 2017

Change https://golang.org/cl/54871 mentions this issue: cmd/internal/obj/x86: don't apply workaround for solaris to darwin

gopherbot pushed a commit that referenced this issue Aug 17, 2017

cmd/internal/obj/x86: don't apply workaround for solaris to darwin
Currently, we have a workaround for solaris that enforce aboslute
addressing for external symbols. However, We don't want to use the
workaround for darwin.
This CL also refactors code a little bit, because the original function
name is not appropriate now.

Updates #17490

Change-Id: Id21f9cdf33dca6a40647226be49010c2c324ee24
Reviewed-on: https://go-review.googlesource.com/54871
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 22, 2017

Despite the release-blocker label I don't see this happening for Go 1.10. Moving to Go 1.11. If we really want to do this we need to make sure work starts at the beginning of the cycle.

@rsc rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Nov 22, 2017

Nick Kledzik suggested in e-mail:

There is a middle ground that some JIT creators use, which is to wrap their JITed code in a “boilerplate” mach-o.

For the Go case, since you already generate the wrapper mach-o structure for executables, you expand that a bit, and standard content for linking to libSystem.dylib. This would include the stubs (PLT) section and lazy pointer section (GOT), and the LINKEDIT stuff to bind them for all the POSIX calls Go supports. The Go compiler then calls the stub (PLT) instead of directly inlining the syscall.

The new mach-o content would be the same for all Go executables, so you don’t need a linker. You write a C program that calls all POSIX functions Go supports and compile/link that once, and copy the stubs/lazy pointer sections and LINKEDIT content for use as your boilerplate.

This won’t solve the thread local issue, but will make Go executables true dynamic executables and free Apple to evolve the private libSystem/kernel syscall interface.

@gopherbot

This comment has been minimized.

Copy link

commented Oct 11, 2018

Change https://golang.org/cl/141639 mentions this issue: syscall: implement syscalls on Darwin using libSystem

gopherbot pushed a commit that referenced this issue Nov 7, 2018

syscall: implement syscalls on Darwin using libSystem
There are still some references to the bare Syscall functions
in the stdlib. I will root those out in a following CL.
(This CL is big enough as it is.)
Most are in vendor directories:

cmd/vendor/golang.org/x/sys/unix/
vendor/golang_org/x/net/route/syscall.go
syscall/bpf_bsd.go
syscall/exec_unix.go
syscall/flock.go

Update #17490

Change-Id: I69ab707811530c26b652b291cadee92f5bf5c1a4
Reviewed-on: https://go-review.googlesource.com/c/141639
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Elias Naur <elias.naur@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Nov 7, 2018

Change https://golang.org/cl/148257 mentions this issue: syscall: move uses of Syscall to libSystem on darwin

gopherbot pushed a commit that referenced this issue Nov 8, 2018

syscall: move uses of Syscall to libSystem on darwin
Miscellaneous additional conversions from raw syscalls
to using their libc equivalent.

Update #17490

Change-Id: If9ab22cc1d676c1f20fb161ebf02b0c28f71585d
Reviewed-on: https://go-review.googlesource.com/c/148257
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 10, 2018

@randall77 Any reason to keep this issue open?

@randall77

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2018

Yes, I still need to do x/sys/unix and vendor it into the stdlib.
I think everything else is done.

@gopherbot

This comment has been minimized.

Copy link

commented Dec 14, 2018

Change https://golang.org/cl/154179 mentions this issue: x/sys/unix: convert Darwin syscalls from raw to libSystem

@gopherbot

This comment has been minimized.

Copy link

commented Dec 18, 2018

Change https://golang.org/cl/154660 mentions this issue: unix: remove raw syscall from Getfsstat

@gopherbot

This comment has been minimized.

Copy link

commented Dec 18, 2018

Change https://golang.org/cl/154661 mentions this issue: unix: remove raw syscall from getattrlistTimes

@gopherbot

This comment has been minimized.

Copy link

commented Dec 18, 2018

Change https://golang.org/cl/154662 mentions this issue: unix: remove raw syscall from getAttrList

@gopherbot

This comment has been minimized.

Copy link

commented Dec 18, 2018

Change https://golang.org/cl/154663 mentions this issue: unix: remove raw syscall from Sendfile

gopherbot pushed a commit to golang/sys that referenced this issue Dec 20, 2018

unix: convert Darwin syscalls from raw to libSystem
Call into libSystem (Darwin's libc equivalent) to do system calls.
Raw syscalls are not a supported ABI on Darwin, they want us to use
libSystem calls instead.

The stdlib has already been converted to this new regime.  This
package is vendored into the stdlib, so we need to fix it also, then
vendor it in again.

Because this package isn't tied to a Go release, we need to keep the
old code around and use the new code only for 1.12 and up.

Update golang/go#17490

Change-Id: Idbcacff096b5bfeac871aa75dfd971570ac93322
Reviewed-on: https://go-review.googlesource.com/c/154179
Reviewed-by: Ian Lance Taylor <iant@golang.org>

gopherbot pushed a commit to golang/sys that referenced this issue Dec 20, 2018

unix: remove raw syscall from Getfsstat
Use libc version instead.

Update golang/go#17490

Change-Id: Ibcedccb9715961904d12fc85f69a7139b6c26658
Reviewed-on: https://go-review.googlesource.com/c/154660
Reviewed-by: Ian Lance Taylor <iant@golang.org>

gopherbot pushed a commit to golang/sys that referenced this issue Dec 20, 2018

unix: remove raw syscall from getattrlistTimes
Update golang/go#17490

Change-Id: I29feed5ddea976b39bd4c43bd1ff5942f47df083
Reviewed-on: https://go-review.googlesource.com/c/154661
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot pushed a commit to golang/sys that referenced this issue Dec 20, 2018

unix: remove raw syscall from getAttrList
Update golang/go#17490

Change-Id: I55ea10ce2eb5fb1c0518a57900e78e5f0a29b893
Reviewed-on: https://go-review.googlesource.com/c/154662
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot pushed a commit to golang/sys that referenced this issue Dec 20, 2018

unix: remove raw syscall from Sendfile
Update golang/go#17490

Change-Id: Iaec54b8ffda1a24d4c8b5671185d570fb8683155
Reviewed-on: https://go-review.googlesource.com/c/154663
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Dec 22, 2018

Change https://golang.org/cl/155737 mentions this issue: cmd: vendor x/sys/unix into the stdlib

@gopherbot gopherbot closed this in 55e3ace Dec 25, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Jan 6, 2019

Change https://golang.org/cl/156346 mentions this issue: unix: make Fcntl* routines use libSystem on Darwin

gopherbot pushed a commit to golang/sys that referenced this issue Jan 6, 2019

unix: make Fcntl* routines use libSystem on Darwin
Missed this one.

Update golang/go#17490.

Change-Id: I5ab2197f9981b712b37d3060f72209bebcadf291
Reviewed-on: https://go-review.googlesource.com/c/156346
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Jan 6, 2019

Change https://golang.org/cl/156365 mentions this issue: cmd: vendor x/sys/unix into the stdlib

gopherbot pushed a commit that referenced this issue Jan 7, 2019

cmd: vendor x/sys/unix into the stdlib
upstream git hash: 1775db3f06b568179d273425900dd09125831dd5

Update #17490

Change-Id: I95e3c57137756c5c7a9b7334075caef66f205231
Reviewed-on: https://go-review.googlesource.com/c/156365
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

aykevl added a commit to tinygo-org/tinygo that referenced this issue Mar 24, 2019

darwin: use custom syscall pkg that uses libsystem
Go 1.12 switched to using libSystem.dylib for system calls, because
Apple recommends against doing direct system calls that Go 1.11 and
earlier did. For more information, see:
  golang/go#17490
  https://developer.apple.com/library/archive/qa/qa1118/_index.html

While the old syscall package was relatively easy to support in TinyGo
(just implement syscall.Syscall*), this got a whole lot harder with Go
1.12 as all syscalls now go through CGo magic to call the underlying
libSystem functions. Therefore, this commit overrides the stdlib syscall
package with a custom package that performs calls with libc (libSystem).
This may be useful not just for darwin but for other platforms as well
that do not place the stable ABI at the syscall boundary like Linux but
at the libc boundary.

Only a very minimal part of the syscall package has been implemented, to
get the tests to pass. More calls can easily be added in the future.

aykevl added a commit to tinygo-org/tinygo that referenced this issue Mar 24, 2019

darwin: use custom syscall pkg that uses libsystem
Go 1.12 switched to using libSystem.dylib for system calls, because
Apple recommends against doing direct system calls that Go 1.11 and
earlier did. For more information, see:
  golang/go#17490
  https://developer.apple.com/library/archive/qa/qa1118/_index.html

While the old syscall package was relatively easy to support in TinyGo
(just implement syscall.Syscall*), this got a whole lot harder with Go
1.12 as all syscalls now go through CGo magic to call the underlying
libSystem functions. Therefore, this commit overrides the stdlib syscall
package with a custom package that performs calls with libc (libSystem).
This may be useful not just for darwin but for other platforms as well
that do not place the stable ABI at the syscall boundary like Linux but
at the libc boundary.

Only a very minimal part of the syscall package has been implemented, to
get the tests to pass. More calls can easily be added in the future.

aykevl added a commit to tinygo-org/tinygo that referenced this issue Apr 4, 2019

darwin: use custom syscall pkg that uses libsystem
Go 1.12 switched to using libSystem.dylib for system calls, because
Apple recommends against doing direct system calls that Go 1.11 and
earlier did. For more information, see:
  golang/go#17490
  https://developer.apple.com/library/archive/qa/qa1118/_index.html

While the old syscall package was relatively easy to support in TinyGo
(just implement syscall.Syscall*), this got a whole lot harder with Go
1.12 as all syscalls now go through CGo magic to call the underlying
libSystem functions. Therefore, this commit overrides the stdlib syscall
package with a custom package that performs calls with libc (libSystem).
This may be useful not just for darwin but for other platforms as well
that do not place the stable ABI at the syscall boundary like Linux but
at the libc boundary.

Only a very minimal part of the syscall package has been implemented, to
get the tests to pass. More calls can easily be added in the future.

deadprogram added a commit to tinygo-org/tinygo that referenced this issue Apr 5, 2019

darwin: use custom syscall pkg that uses libsystem
Go 1.12 switched to using libSystem.dylib for system calls, because
Apple recommends against doing direct system calls that Go 1.11 and
earlier did. For more information, see:
  golang/go#17490
  https://developer.apple.com/library/archive/qa/qa1118/_index.html

While the old syscall package was relatively easy to support in TinyGo
(just implement syscall.Syscall*), this got a whole lot harder with Go
1.12 as all syscalls now go through CGo magic to call the underlying
libSystem functions. Therefore, this commit overrides the stdlib syscall
package with a custom package that performs calls with libc (libSystem).
This may be useful not just for darwin but for other platforms as well
that do not place the stable ABI at the syscall boundary like Linux but
at the libc boundary.

Only a very minimal part of the syscall package has been implemented, to
get the tests to pass. More calls can easily be added in the future.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.