Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled #48112

Closed
igalic opened this issue Sep 1, 2021 · 18 comments
Closed

cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled #48112

igalic opened this issue Sep 1, 2021 · 18 comments
Labels
Milestone

Comments

@igalic
Copy link

@igalic igalic commented Sep 1, 2021

What version of Go are you using (go version)?

$ go version
go version go1.17 freebsd/amd64

Does this issue reproduce with the latest release?

This is the latest version.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/meena/.cache/go-build"
GOENV="/home/meena/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOINSECURE=""
GOMODCACHE="/home/meena/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="freebsd"
GOPATH="/home/meena/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build337677643=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I compiled a simple hello world program:

package main

import (
        "fmt"
)

func main() {
        fmt.Println("Hello, World!")
}

, and tried to run it in a FreeBSD 13 environment with W^X enabled:

meena@bsdix ~/s/wxorx> go build hello.go
meena@bsdix ~/s/wxorx> sudo -H sysctl kern.elf64.allow_wx=0
kern.elf64.allow_wx: 1 -> 0
meena@bsdix ~/s/wxorx>

What did you expect to see?

meena@bsdix ~/s/wxorx> ./hello
Hello, World!
meena@bsdix ~/s/wxorx>

What did you see instead?

meena@bsdix ~/s/wxorx> ./hello
exec_new_vmspace: mapping stack size 0x20000000 prot 0x7 failed mach error 2 errno 13
fish: Job 1, './hello' terminated by signal SIGABRT (Abort)
meena@bsdix ~/s/wxorx [SIGABRT]>

This also happens if you run any go program:

meena@bsdix ~/s/wxorx [SIGABRT]> go version
exec_new_vmspace: mapping stack size 0x20000000 prot 0x7 failed mach error 2 errno 13
fish: Job 1, 'go version' terminated by signal SIGABRT (Abort)
meena@bsdix ~/s/wxorx [SIGABRT]>

Data Execution Prevention Windows is supported since go 1.14:
https://golang.org/doc/go1.14#windows "Go binaries on Windows now have DEP (Data Execution Prevention) enabled."

I also don't hear any complaints from OpenBSD people, who have W^X enabled for about 20 years now.

Note that workarounds of marking a binary as wxneeded with elfctl(1) also don't work:

meena@bsdix ~/s/wxorx> elfctl -e +wxneeded hello
elfctl: NT_FREEBSD_FEATURE_CTL note not found
elfctl: NT_FREEBSD_FEATURE_CTL note not found
meena@bsdix ~/s/wxorx [1]>

Somebody managed to extract that "Note" and link it against their rclone: https://forums.freebsd.org/threads/rclone-not-working-with-w-x.80279/ which allowed them to work around this.
I have not yet been able to reproduce that work-around.

@igalic
Copy link
Author

@igalic igalic commented Sep 1, 2021

Addendum: With go build -buildmode=pie the resulting binary works!

meena@bsdix ~/s/wxorx> go build -buildmode=pie hello.go
meena@bsdix ~/s/wxorx> ./hello
Hello, World!
meena@bsdix ~/s/wxorx> sudo -H sysctl kern.elf64.allow_wx=0
Password:
kern.elf64.allow_wx: 1 -> 0
meena@bsdix ~/s/wxorx> ./hello
Hello, World!
meena@bsdix ~/s/wxorx> go version
exec_new_vmspace: mapping stack size 0x20000000 prot 0x7 failed mach error 2 errno 13
fish: Job 1, 'go version' terminated by signal SIGABRT (Abort)
meena@bsdix ~/s/wxorx [SIGABRT]>

not only that, elfctl is happy with the binary:

meena@bsdix ~/s/wxorx> elfctl hello
File 'hello' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user VA to 48bit' is unset.
noaslrstkgap    'Disable ASLR stack gap' is unset.
meena@bsdix ~/s/wxorx>

Loading

@emaste
Copy link

@emaste emaste commented Sep 1, 2021

kib@ explained at least part of the issue: -buildmode=exe results in a binary with no PT_GNU_STACK, so defaults to rwx stack and thus fails with allow_wx=0

It seems -buildmode=pie causes go to use (parts of) the standard host tool chain, and so gets both a PT_GNU_STACK for rw stack, and the NT_FREEBSD_FEATURE_CTL note that elfctl manipulates.

Loading

@igalic
Copy link
Author

@igalic igalic commented Sep 1, 2021

We can modify our ports system to set -buildmode=pie for ports depending on go.
That means, we can produce packages, built with go, that will be compiled as PIEs.

However, we cannot seem to be able to produce a go compiler that is PIE.

Put differently: we can build packages with go on a system with WX disabled, that will work on a system with WX enabled, but we cannot produce a go compiler that will work on such a system.

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Sep 1, 2021

We already set PT_GNU_STACK for Linux, we just need to do so for freebsd also (and all the other bsds?). That would solve the problem for internal linking, and I think that's the situation when it is happening?

Loading

@emaste
Copy link

@emaste emaste commented Sep 1, 2021

Yes, we need PT_GNU_STACK on FreeBSD. I believe both OpenBSD and NetBSD provide a no-X stack in any case and would be fine without PT_GNU_STACK (although it might not cause any trouble to have it).

That would solve the problem for internal linking

I believe so, yes. I'm not completely familiar with go's linking/tool chain use, but I assume that -buildmode=pie ends up relying on the standard host linker as a side effect? (And, this may be an implementation detail that could change in the future?)

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 1, 2021

Change https://golang.org/cl/346872 mentions this issue: cmd/link: mark stacks as non-executable on freebsd

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Sep 1, 2021

@igalic @emaste Try the above linked CL and tell me if that works for you.

Loading

@emaste
Copy link

@emaste emaste commented Sep 1, 2021

CC @dmgk, FreeBSD's lang/go port maintainer

Loading

@cherrymui cherrymui changed the title Unable to execute go binaries on a FreeBSD system with W^X enabled cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled Sep 1, 2021
@cherrymui cherrymui added this to the Go1.18 milestone Sep 1, 2021
@igalic
Copy link
Author

@igalic igalic commented Sep 2, 2021

with that patch it now works with w^x enabled:

meena@bsdix /u/p/l/go (main)> file /usr/local/go/bin/go*
/usr/local/go/bin/go:    ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, Go BuildID=XLIT5y7zsZF5lR-GsGHn/5sUPdtoWTSvCjrig1rac/oUGhGkYcwZr5aypGE13h/PyCGVZmPPahQmZyis-c2, not stripped
/usr/local/go/bin/gofmt: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, Go BuildID=23dNAVgDQa9cM8y7STWR/ZMhOBGxd00N-4hKWF1zI/-MZO7QHiu4q61cHkSr4w/107R6PUcs881Y9-en7pT, not stripped
meena@bsdix /u/p/l/go (main)> elfctl /usr/local/go/bin/go*
elfctl: NT_FREEBSD_FEATURE_CTL note not found
elfctl: NT_FREEBSD_FEATURE_CTL note not found
meena@bsdix /u/p/l/go (main) [1]> sudo -H sysctl kern.elf64.allow_wx=0
kern.elf64.allow_wx: 1 -> 0
meena@bsdix /u/p/l/go (main)> go version
go version go1.17 freebsd/amd64
meena@bsdix /u/p/l/go (main)>

(this means that whenever the bootstrap compiler switches to Go1.18, we could start building lang/go on w^x enabled systems)

Loading

@gopherbot gopherbot closed this in 782aa42 Sep 2, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Sep 2, 2021

Do we want to backport this, so Go 1.16 and 1.17 will work on newer FreeBSD? Thanks.

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Sep 2, 2021

I guess that depends on how often people enable W^X mode. My intuition is not very often, or we'd have seen this before.
Is there a plan to make W^X the default?

Loading

@emaste
Copy link

@emaste emaste commented Sep 2, 2021

My intuition is not very often, or we'd have seen this before. Is there a plan to make W^X the default?

You're right that it's not enabled often at present. Right now it's really "adventurous" users trying it out; I expect we will make a more comprehensive effort within the FreeBSD developer community to find showstoppers (e.g. major runtimes that fail with it enabled, as here), before a broader call for testing and any discussion of defaults.

That said, independent of W^X with no PT_GNU_STACK we get a RWX stack, and IMO it's worth backporting this just to avoid that.

Loading

@igalic
Copy link
Author

@igalic igalic commented Sep 2, 2021

@ cherrymui Our "quarterly" branch currently has 1.16 and our "latest" branch has 1.17, so i think it would make sense to backport it to both of those.

I'll also open an issue to have NT_FREEBSD_FEATURE_CTL note added, as @emaste mentioned in the review for https://golang.org/cl/346872
I see that NetBSD and OpenBSD already have a .note section, so there's precedent.

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Sep 2, 2021

I don't think this qualifies for backporting. https://github.com/golang/go/wiki/MinorReleases
It's always been this way, so it isn't a regression. A workaround is to just turn W^X mode off.
It's not really a security issue either - at least not something exploitable (and not new). In fact, I think this would only affect the single stack the program starts with - for freebsd all the other stacks (both G and M stacks) are allocated by the Go runtime and are already not marked executable.

Loading

@igalic
Copy link
Author

@igalic igalic commented Sep 3, 2021

Aye.
I've submitted this as patch to our lang/go port then: https://bugs.freebsd.org/258241 /cc @dmgk

Loading

@yzgyyang
Copy link

@yzgyyang yzgyyang commented Nov 9, 2021

We can modify our ports system to set -buildmode=pie for ports depending on go. That means, we can produce packages, built with go, that will be compiled as PIEs.

However, we cannot seem to be able to produce a go compiler that is PIE.

Any specific blockers on why we cannot have a go compiler that is PIE? Currently, go itself builds fail if set kern.elf64.allow_wx=0.

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 9, 2021

I don't know why the Go tools would fail with allow_wx=0. Can you provide more details?

You should be able to build the Go tools with pie by running something like go install -buildmode=pie cmd.

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 9, 2021

Oh, I see, the problem is the missing PT_GNU_STACK. But that problem should be fixed for 1.18.

If you still have problems on tip, please open a new bug. Thanks.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants