Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: stop using direct syscalls on OpenBSD #36435

Open
4a6f656c opened this issue Jan 7, 2020 · 75 comments
Open

all: stop using direct syscalls on OpenBSD #36435

4a6f656c opened this issue Jan 7, 2020 · 75 comments

Comments

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Jan 7, 2020

Upcoming changes to the OpenBSD kernel will prevent system calls from being made unless they are coming from libc.so (with some exceptions, for example, a static binary). There are also likely to be changes to the APIs used for system calls. As such, the Go runtime (and other packages) need to stop using direct syscalls, rather calling into libc functions instead (as has always been done for Solaris and now also for macOS). This will avoid these issues and allow Go to continue to function correctly on OpenBSD.

A version of Go with openbsd/amd64 being modified to perform all calls via libc is available at:

https://github.com/4a6f656c/go/tree/openbsd-syscall

Further work is still required to convert openbsd/386, openbsd/arm and openbsd/arm64.

@randall77
Copy link
Contributor

@randall77 randall77 commented Jan 7, 2020

with some exceptions, for example, a static binary

But Go fits squarely within that exception. Or are you worried about nonstandard build modes?

@bradfitz bradfitz added this to the Go1.15 milestone Jan 7, 2020
@FiloSottile
Copy link
Contributor

@FiloSottile FiloSottile commented Jan 7, 2020

That exception exists specifically for Go. https://lwn.net/Articles/806863/

@beoran
Copy link

@beoran beoran commented Jan 8, 2020

I think this move to make kernels callable only though a blessed C library is a bad idea, in particular for all users of programming languages that are not C-like. It introduces a performance overhead for every kernel call that could be avoided by direct kernel calls. The security benefits are debatable, fix the kernel, not hide the flaws and incompatibilities behind a C library.

Kernels should do like Linux does, and provide a safe, stable, kernel API, and not try to shift the interface to C libraries. And even then, graphic on Linux is a disaster because they exactly did that, provide most functionality in a C library, not in the kernel itself. We should protest at least once to mr. De Raadt. against this debatable decision.

@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented Jan 8, 2020

@randall77 - in many cases Go on OpenBSD will generate a dynamically linked executable, unless cgo is disabled:

$ cat n.go
package main

import (
        "fmt"
        "net"
)

func main() {
        fmt.Println(net.IPv4len)
}
$ go build -o n n.go 
$ ldd n              
n:
        Start            End              Type  Open Ref GrpRef Name
        0000000000010000 00000000001d3000 exe   2    0   0      n
        0000000453cfd000 0000000453d3e000 rlib  0    1   0      /usr/lib/libpthread.so.26.1
        000000048614c000 000000048624c000 rlib  0    1   0      /usr/lib/libc.so.95.1
        0000000497c90000 0000000497c90000 ld.so 0    1   0      /usr/libexec/ld.so
$ CGO_ENABLED=0 go build -o n n.go
ldd n
$ ldd n
n:
not a dynamic executable

This would mean that we'd have to drop cgo support and only support statically compiled Go executables.

@FiloSottile you're correct in that there is currently an exception for dynamically linked executables to allow syscalls from both libc.so and the main text segment, so that Go continues to work. However, in addition to this there will always likely be an exception for static binaries, since in that case there is no libc.so and parts of libc.a have likely been linked into the main text segment (otherwise a static binary could not make syscalls).

@ghost
Copy link

@ghost ghost commented Jan 12, 2020

@4a6f656c why does it generate dynamically linked executable in the first place?

@gopherbot
Copy link

@gopherbot gopherbot commented May 20, 2020

Change https://golang.org/cl/234381 mentions this issue: runtime, syscall: correct openbsd/arm and openbsd/arm64 syscalls for OpenBSD 6.7

gopherbot pushed a commit that referenced this issue May 26, 2020
…OpenBSD 6.7

Add two no op instructions following svc on openbsd/arm64 and swi on openbsd/arm.

All except some of the most recent arm64 processors have a speculative execution
flaw that occurs across a syscall boundary, which cannot be mitigated in the
kernel. In order to protect against this leak a speculation barrier needs to be
placed after an svc or swi instruction.

In order to avoid the performance impact of these instructions, the OpenBSD 6.7
kernel returns execution two instructions past the svc or swi call. For now two
hardware no ops are added, which allows syscalls to work with both 6.6 and 6.7.
These should be replaced with real speculation barriers once OpenBSD 6.8 is
released.

Updates #36435

Change-Id: I06153cb0998199242cca8761450e53599c3e7de4
Reviewed-on: https://go-review.googlesource.com/c/go/+/234381
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 16, 2020

Moving to backlog milestone.

@bw-86
Copy link

@bw-86 bw-86 commented Jul 17, 2020

I would like to work on this. After a brief look, Solaris seems to be a good role model for OpenBSD. What would be the best strategy here? OpenBSD seems to share definitions with the other BSD platforms, which result in conflicting definitions when I try to hook it into mksyscall_libc.pl.
Or should I try to do things more like MacOS? Is that an easier way? How should I proceed?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 17, 2020

I don't think there is any fundamental difference between the Solaris approach and the macOS approach. Following the Solaris approach seems fine if that is easier.

Yes, you will undoubtedly have to shuffle the build tags to separate openbsd from the other BSD systems in some cases.

@bw-86
Copy link

@bw-86 bw-86 commented Jul 18, 2020

Well, I started. My WIP commits will be pushed here, if anyone is interested in them: https://github.com/bw-86/go

Currently, nothing is building, of course. I am chasing from one linking error to the next one while imitating what Solaris does. I will notify you when I'm getting somewhere (or when I have questions I cannot answer myself).

EDIT: I changed the default branch of my fork to 4a6f656c's work. My branch is openbsd-syscalls (plural), if anyone wants to see what I tried myself.

@ghost
Copy link

@ghost ghost commented Jul 18, 2020

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

I agree with @beoran and iirc OpenBSD will not enforce libc.so stubs if you won't play their security theater, so you don't have to sacrifice both efficiency and actual security.

@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented Jul 18, 2020

I would like to work on this. After a brief look, Solaris seems to be a good role model for OpenBSD. What would be the best strategy here? OpenBSD seems to share definitions with the other BSD platforms, which result in conflicting definitions when I try to hook it into mksyscall_libc.pl.
Or should I try to do things more like MacOS? Is that an easier way? How should I proceed?

@bw-86, the git repo referenced in the first comment contains a full implementation for openbsd/amd64:

https://github.com/4a6f656c/go/tree/openbsd-syscall

I've since rebased and cleaned some aspects of this up, but need to push a new branch.

As noted, there is still work required to complete the migration for openbsd/386, openbsd/arm and openbsd/arm64. There are also some issues relating to the linker that impact this work (see #39257 and #39256). Aside from the remaining architectures, most of the remaining work is around being able to split this up to merge upstream and I have some concerns about cross-compiling and bootstrapping that I've not yet had time to investigate.

@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented Jul 18, 2020

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

Go will always use dynamic linking (and possibly always external linking) - it would be a significant amount of effort to maintain both libc based syscalls and raw syscalls and switch between to two. Any Go binary on OpenBSD that pulls in the net package is already going to result in a dynamic cgo-based binary (unless cgo is explicitly disabled - see #36435 (comment)).

I agree with @beoran and iirc OpenBSD will not enforce libc.so stubs if you won't play their security theater, so you don't have to sacrifice both efficiency and actual security.

If you do not want OpenBSD's "security theater" then I would suggest not using OpenBSD. Additionally, have you actually measured the performance difference? Both Solaris and macOS already both use libc-style syscalls and IIRC the switch on macOS resulted in a minimal performance impact.

@bw-86
Copy link

@bw-86 bw-86 commented Jul 18, 2020

Heh, how could I miss that? Well, at least I learned something by trying to do it myself.

@4a6f656c: Can I do something to accelerate this? Should I try to adapt your work to i386? I have a system here were I could develop and test it.

@ghost

This comment was marked as disruptive content.

@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented Jul 18, 2020

Heh, how could I miss that? Well, at least I learned something by trying to do it myself.

That's always a good thing :)

@4a6f656c: Can I do something to accelerate this? Should I try to adapt your work to i386? I have a system here were I could develop and test it.

Getting it working on openbsd/386 would be great!

I've just pushed an update to that repo that is based on Go -tip.

@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented Jul 18, 2020

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

Go will always use dynamic linking (and possibly always external linking) - it would be a significant amount of effort to maintain both libc based syscalls and raw syscalls and switch between to two. Any Go binary on OpenBSD that pulls in the net package is already going to result in a dynamic cgo-based binary (unless cgo is explicitly disabled - see #36435 (comment)).

Actually, there is another thing that I need to investigate - in many cases we should be able to statically link against libc.a, which would actually allow us to produce static Go binaries in more cases then we currently do on OpenBSD.

@jrick
Copy link
Contributor

@jrick jrick commented Jul 18, 2020

I can chime in as both an OpenBSD user and someone who works on a project that builds releases for OpenBSD. I am happy with this change but hope that cross compiling to OpenBSD and reproducible builds remain possible. This probably negates requiring the external linker and having libc.a available.

The current behavior for macOS is ideal in this case. When cross compiling, it is still able to create a dynamically linked executable, and does so reproducibly, without depending on the external macOS linker.

$ uname
OpenBSD
$ cat importnet.go                                                             
package main

import _ "net"

func main() {}
$ CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build importnet.go 
$ echo put importnet | sftp machost
Connected to machost.
sftp> put importnet
Uploading importnet to /Users/jrick/importnet
importnet                                     100% 2047KB   2.6MB/s   00:00    
$ ssh machost './importnet && otool -L importnet'
importnet:
        /usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 0.0.0, current version 0.0.0)
@bw-86
Copy link

@bw-86 bw-86 commented Jul 19, 2020

I have to admit defeat for the moment. I pushed some minor work to my repository; feel free to take it.
I will have to learn some more assembly and more about the calling conventions of Go and OpenBSD before I can tackle this. To be precise, I don't know how to write the syscall functions (syscall6, syscall9, ...).

So, can someone help me with this one? If I had just one of them, I might adapt it for the other use cases.

@golang golang deleted a comment Jul 21, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 24, 2020

Change https://golang.org/cl/250180 mentions this issue: cmd/dist,cmd/link,runtime: switch openbsd/amd64 to pthreads

@cherrymui cherrymui modified the milestones: Go1.16, Go1.17 Feb 8, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Feb 8, 2021

Thanks @odeke-em. Moved to 1.17. There are still 386, ARM (32-bit), MIPS64.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 21, 2021

Change https://golang.org/cl/312229 mentions this issue: syscall: use libc in Exec on openbsd/arm64

gopherbot pushed a commit that referenced this issue Apr 21, 2021
Like on openbsd/amd64, use libc instead of direct syscalls on
openbsd/arm64.

This was likely missed in CL 286815. A similar change was done for
openbsd/amd64 in CL 270380.

Updates #36435

Change-Id: Ie496a6130f1a43d30974502777db12217e65c551
Reviewed-on: https://go-review.googlesource.com/c/go/+/312229
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Joel Sing <joel@sing.id.au>
@ianlancetaylor ianlancetaylor modified the milestones: Go1.17, Backlog Apr 22, 2021
@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented Apr 22, 2021

@ianlancetaylor @cherrymui is there any way that we are able to get some traction on this before we reach the end of the current development cycle?

The changes for openbsd/386 were sent out during the previous development cycle and are still pending review. I also have openbsd/arm changes that I was holding until the openbsd/386 changes landed (in order to try to reduce the amount of tangle).

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Apr 22, 2021

Reviewing the 386 CL is still on my list. It is not forgotten. I'll do it as soon as I can. The register ABI stuff has been keeping me occupied, but as it is wrapping up I think I'll have some time. Sorry for the delay.

@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented Apr 22, 2021

@cherrymui - no problem, thanks for the update.

gopherbot pushed a commit that referenced this issue Apr 30, 2021
This switches openbsd/386 to thread creation via pthreads, rather than doing
direct system calls.

Update #36435

Change-Id: I000a815fc0edd0272c3285954f3f007229bc60a0
Reviewed-on: https://go-review.googlesource.com/c/go/+/250577
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
gopherbot pushed a commit that referenced this issue Apr 30, 2021
Switch openbsd/386 to locking via libc, rather than performing direct
system calls.

Update #36435

Change-Id: I8198171e21f9acf28846ad723ea9ff48f7c8a69d
Reviewed-on: https://go-review.googlesource.com/c/go/+/287652
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
gopherbot pushed a commit that referenced this issue Apr 30, 2021
Use libc rather than performing direct system calls for the runtime on
openbsd/386.

Updates #36435

Change-Id: I0cd65368bc824c81f5f98ea24e4f82db5468b170
Reviewed-on: https://go-review.googlesource.com/c/go/+/287653
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
gopherbot pushed a commit that referenced this issue Apr 30, 2021
Convert the syscall package on openbsd/386 to use libc rather than performing
direct system calls.

Updates #36435

Change-Id: Ifcfbca0e6b933762596a564243caa850dac01442
Reviewed-on: https://go-review.googlesource.com/c/go/+/287654
Trust: Joel Sing <joel@sing.id.au>
Run-TryBot: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315794 mentions this issue: runtime: skip TestCrashDumpsAllThreads on openbsd/arm

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315790 mentions this issue: runtime: switch openbsd/arm to pthreads

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315793 mentions this issue: runtime,syscall: convert syscall on openbsd/arm to libc

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315791 mentions this issue: runtime: switch openbsd/arm locking to libc

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315792 mentions this issue: runtime: switch runtime to libc for openbsd/arm

gopherbot pushed a commit that referenced this issue May 9, 2021
This switches openbsd/arm to thread creation via pthreads, rather than doing
direct system calls.

Update #36435

Change-Id: Ia8749e3723a9967905c33b6d93dfd9be797a486c
Reviewed-on: https://go-review.googlesource.com/c/go/+/315790
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
gopherbot pushed a commit that referenced this issue May 9, 2021
Switch openbsd/arm to locking via libc, rather than performing direct
system calls.

Update #36435

Change-Id: I190abb1aa544d2cb406fe412960ec106c9716f87
Reviewed-on: https://go-review.googlesource.com/c/go/+/315791
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
gopherbot pushed a commit that referenced this issue May 9, 2021
Use libc rather than performing direct system calls for the runtime on
openbsd/arm.

Updates #36435

Change-Id: If64a96a61c80b9748792f8a85a8f16ed6ebca91f
Reviewed-on: https://go-review.googlesource.com/c/go/+/315792
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
gopherbot pushed a commit that referenced this issue May 9, 2021
Convert the syscall package on openbsd/arm to use libc rather than performing
direct system calls.

Updates #36435

Change-Id: Iff3a91c959292cbf4e0a09c7fd34efc8e88ff83f
Reviewed-on: https://go-review.googlesource.com/c/go/+/315793
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
gopherbot pushed a commit that referenced this issue May 9, 2021
This test is also now flakey on this platform.

Updates #36435
Updates #42464

Change-Id: Idedb81478178ffffe7a9c125a6e8bbd83458f9ab
Reviewed-on: https://go-review.googlesource.com/c/go/+/315794
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented May 9, 2021

Change https://golang.org/cl/317919 mentions this issue: runtime,syscall: simplify openbsd related build tags

gopherbot pushed a commit that referenced this issue May 11, 2021
openbsd/mips64 is now the only openbsd port that uses non-libc syscall - revise
build tags to reflect this.

Update #36435

Change-Id: I357b2dd2926d058e25e618fcca42c388587598a8
Reviewed-on: https://go-review.googlesource.com/c/go/+/317919
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented May 11, 2021

@ianlancetaylor @cherrymui - given that misp64 lacks internal linking support for cgo, I took a look at switching openbsd/mips64 to external-only linking, which would allow for it to be converted to libc syscalls. However, adding openbsd/mips64 to cmd/internal/sys/supported.MustLinkExternal results in:

$ GOROOT_BOOTSTRAP=/home/joel/go-openbsd-mips64-bootstrap ./make.bash  
Building Go cmd/dist using /home/joel/go-openbsd-mips64-bootstrap. (devel go1.17-c14ecaca81 Sun May 9 17:07:22 2021 +0000 openbsd/mips64)
Building Go toolchain1 using /home/joel/go-openbsd-mips64-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
warning: unable to find runtime/cgo.a
Building Go toolchain2 using go_bootstrap and Go toolchain1.                                                      
# cmd/asm                                                                                                         
loadinternal: cannot find runtime/cgo                                                                             
# cmd/cgo                                                                                                         
loadinternal: cannot find runtime/cgo
# cmd/compile                                                                                                     
loadinternal: cannot find runtime/cgo                                                                                                                                                                                               
# cmd/link                                                                                                        
loadinternal: cannot find runtime/cgo                                                                             
Building Go toolchain3 using go_bootstrap and Go toolchain2.
# cmd/asm                                                                                                         
loadinternal: cannot find runtime/cgo                                                                             
# cmd/cgo                                                                                                         
loadinternal: cannot find runtime/cgo
# cmd/compile                                                                                                     
loadinternal: cannot find runtime/cgo                                                                             
# cmd/link
loadinternal: cannot find runtime/cgo
Building packages and commands for openbsd/mips64.
# cmd/addr2line
loadinternal: cannot find runtime/cgo
# cmd/api
loadinternal: cannot find runtime/cgo
# cmd/asm
...

Everything appears to build and work correctly (although I've not yet tried switching to libc syscalls) - any ideas on investigating/fixing loadinternal: cannot find runtime/cgo? An interesting point is that running various tests (e.g. go test path) result in the same error, until I clear the Go cache - after that it builds and runs without the error...

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented May 11, 2021

This is probably due to #31544.

If the programs can be built correctly and the only issue is the warning prints, that is probably fine. We have that in the past for some mobile platforms as well, which also require external linking. Maybe you want to add a case to cmd/go/internal/load/pkg.go:externalLinkingForced.

@4a6f656c
Copy link
Contributor Author

@4a6f656c 4a6f656c commented May 11, 2021

This is probably due to #31544.

If the programs can be built correctly and the only issue is the warning prints, that is probably fine. We have that in the past for some mobile platforms as well, which also require external linking. Maybe you want to add a case to cmd/go/internal/load/pkg.go:externalLinkingForced.

Ah ha! This is likely the bit that grep was failing to find - thanks, trying that now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
14 participants