Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: build: add -static flag #26492

Open
kolyshkin opened this issue Jul 19, 2018 · 26 comments
Open

cmd/go: build: add -static flag #26492

kolyshkin opened this issue Jul 19, 2018 · 26 comments

Comments

@kolyshkin
Copy link
Contributor

@kolyshkin kolyshkin commented Jul 19, 2018

This is a proposal to add -static flag to go build.

Producing a static build with go already requires a non-trivial amount of flags passed to go build, which on Linux currently amounts to something like:

-ldflags '-extldflags "-fno-PIC -static"' -buildmode pie -tags 'osusergo netgo static_build' [1]

...and this magic string keeps growing.

It would be awesome to encapsulate this sacred knowledge internally, exposing it via a new -static flag for go build and friends.

In addition to the above benefit of hiding the complexity, -static can also bring us more niceties, such as:

  • automatic static build tag third-party software can rely upon. Currently using static_build tag seems like a de-facto standard, used a lot, but it has to be explicitly defined like in [1] above.
  • providing --static flag to pkg-config invocations (those initiated by // #cgo pkg-config: lib lines in the source code). It will solve another issue (linking against a proper set of libraries for both static and dynamic link cases) for which a somewhat verbose workaround is currently required (for example, see ploop_link_static.go and ploop_link_dynamic.go).

See also

  • #12058 (original proposal for adding conditional --static to pkg-config)
  • #24787 (initial version of this very proposal)
@ianlancetaylor ianlancetaylor changed the title [proposal] go build: add -static flag proposal: cmd/go: build: add -static flag Jul 20, 2018
@gopherbot gopherbot added this to the Proposal milestone Jul 20, 2018
@rsc
Copy link
Contributor

@rsc rsc commented Jul 23, 2018

This seems OK but the tag should just be "static" not "static_build" (it's a build tag already). And we should make os/user and net do the right thing - whatever that is - instead of having to hard-code their tags too.

Note that the assumption is that -static does not disable cgo; it just makes the cgo uses statically linked. FWIW, I don't know how well statically linked cgo works, but apparently well enough.

It sounds like maybe the os/user and net non-cgo restrictions only apply on Linux (more precisely, GNU/Linux, since this is a glibc restriction).

@rsc rsc removed this from the Proposal milestone Jul 23, 2018
@rsc rsc added this to the Go1.12 milestone Jul 23, 2018
@rsc
Copy link
Contributor

@rsc rsc commented Jul 23, 2018

If anyone wants to work on this for Go 1.12, help is appreciated.

@amenzhinsky
Copy link
Contributor

@amenzhinsky amenzhinsky commented Jul 24, 2018

I get the idea of combining netgo and usergo tags that looks a bit magical, but what's the reason of using the pie builmode since it forces the toolchain to use an external linker (gcc, clang) and many build systems, like alpine docker images, come without one.

@kolyshkin
Copy link
Contributor Author

@kolyshkin kolyshkin commented Jul 24, 2018

This seems OK but the tag should just be "static" not "static_build" (it's a build tag already).

static_build was proposed as it is used in many packages (seems like a de facto standard... I'm too lazy to provide examples but please let me know if you need any).

Still, I agree it's better to have static as it is simple and straightforward (and hopefully won't collide with anything). Proposal changed.

And we should make os/user and net do the right thing - whatever that is

Right (and current netgo and osusergo flags might stay for backward compatibility, as they might also be used to choose one implementation over another, with no regard to static/dynamic linking).

Note that the assumption is that -static does not disable cgo; it just makes the cgo uses statically linked.

Great.

many build systems, like alpine docker images, come without one

I'm afraid I was overly brief describing the idea, let me emphasize.

The idea (specifically, the first part of it) is to hide the burden of choosing the build flags required in order to obtain a working static binary (on a best effort/knowledge basis). The above example is particularly valid for Linux/glibc. In case of Linux/musl (i.e. Alpine), for example, there is probably no need to use osusergo or netgo tags.

@iamoryanmoshe
Copy link
Contributor

@iamoryanmoshe iamoryanmoshe commented Jul 24, 2018

Any chance you elaborate on what a static build does differently than regular build?

I would like to work on this proposal but I need further clarification.

@kolyshkin
Copy link
Contributor Author

@kolyshkin kolyshkin commented Jul 25, 2018

Any chance you elaborate on what a static build does differently than regular build?

It should

  1. Set the needed build options (this is platform-dependent, Linux/glibc example is above).
  2. Set the needed build tags (for Linux/glibc this currently amounts to osusergo netgo).
  3. Set the static build tag.
  4. Add the --static flag to pkg-config invocations (those triggered by // #cgo pkg-config: <lib> ... lines in the source code)

@vikramcse
Copy link

@vikramcse vikramcse commented Aug 3, 2018

Can I work on this as my first contribution to go?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 3, 2018

@vikramcse The main requirements here are access to a range of different systems in order to test the work, and the knowledge required to write a good test for whether the program is statically linked. The actual patch is probably not too difficult, I hope. If you still want to give this a try, please go ahead.

@vikramcse
Copy link

@vikramcse vikramcse commented Aug 4, 2018

Thanks @ianlancetaylor, I should start from simpler issue

@tmm1
Copy link
Contributor

@tmm1 tmm1 commented Nov 2, 2018

We build golang binaries across a variety of platforms, preferring static builds wherever possible. Here are the flags we're currently using:

windows: -tags netgo -ldflags '-H=windowsgui -extldflags "-static"'
linux/bsd: -tags netgo -ldflags '-extldflags "-static"'
macos: -ldflags '-s -extldflags "-sectcreate __TEXT __info_plist Info.plist"'
android: -ldflags -s

On macos and android, we need to be able to pull in system frameworks and libraries so we opted not to build static binaries.

@kolyshkin
Copy link
Contributor Author

@kolyshkin kolyshkin commented Nov 6, 2018

@tmm1 thanks for the info! Just noticed you should also use osusergo tag, at least for Linux.

@frickenate
Copy link

@frickenate frickenate commented Jan 16, 2020

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o test -ldflags '-extldflags "-f no-PIC -static"' -buildmode pie -tags 'osusergo netgo static_build' test.go

@ddevault How are you managing to combine CGO_ENABLED=0 and -buildmode pie? I get the error loadinternal: cannot find runtime/cgo, and from what I'm reading elsewhere it seems that buildmode pie is incompatible with disabling cgo? I'm just looking for the smallest static binary I can get, without cgo. If I remove -buildmode pie, it works... but I'm not sure if I should also be removing the -fno-PIC flag as well?

@ddevault
Copy link

@ddevault ddevault commented Jan 16, 2020

loadinternal: cannot find runtime/cgo, as I understand it, is a red herring. I get this when compiling all Go projects under any configuration.

@frickenate
Copy link

@frickenate frickenate commented Jan 16, 2020

Trying to run the binary generated after getting that warning gives me in instant death from a SIGTRAP. Guess I need to choose between -buildmode pie and CGO_ENABLED=0.

@aykevl
Copy link

@aykevl aykevl commented May 1, 2020

[...] In case of Linux/musl (i.e. Alpine), for example, there is probably no need to use osusergo or netgo tags.

For completeness, in #38789 I discovered that osusergo is still necessary on Alpine Linux in some cases.

@zrhoffman
Copy link

@zrhoffman zrhoffman commented May 28, 2020

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o test -ldflags '-extldflags "-f no-PIC -static"' -buildmode pie -tags 'osusergo netgo static_build' test.go

...this is ridiculous

This was sufficient for me to get a statically-linked binary for which readelf -d reports no linkage:

GOOS=linux go build -tags 'osusergo netgo'

Runs fine on CentOS when using CentOS, Alpine Linux, macOS, or Windows as the build OS.

@mark-rushakoff
Copy link
Contributor

@mark-rushakoff mark-rushakoff commented Oct 2, 2020

I'm just mentioning this here in case it helps anyone else with manually configured static builds: when we upgraded to go 1.15, I had to also add -linkmode=external to what we were passing to -ldflags, as mentioned in the 1.15 release notes in the linker section: https://golang.org/doc/go1.15#linker

In our case, we are building static binaries on Debian, to be run on Alpine. Without providing -linkmode=external, attempting to run those applications on Alpine resulted in the obscure error message standard_init_linux.go:211: exec user process caused "no such file or directory". Running file against those binaries noted that they were dynamically linked.

@hazcod
Copy link

@hazcod hazcod commented Oct 2, 2020

@mark-rushakoff ahh nice! You just solved a sleepless night. :-)

@atc0005
Copy link

@atc0005 atc0005 commented Oct 2, 2020

@zrhoffman: This was sufficient for me to get a statically-linked binary for which readelf -d reports no linkage:

GOOS=linux go build -tags 'osusergo netgo'

Do you happen to know whether this only works for Go projects using specific packages, or if it can be relied upon to generate static builds in all cases? I understand the latter is probably too much to hope for considering the length of this thread. :)

EDIT: To answer my own question, no, those build tags alone are not sufficient to produce a working static binary in all cases. CGO appears to make things much more complicated.

atc0005 added a commit to atc0005/check-mail that referenced this issue Oct 3, 2020
The previous build command produces dynamic executables
instead of the intended (but not well noted in the Makefile)
static executables. This commit includes changes noted in
upstream `golang/go` issues which appear to work as intended
for others.

- refs GH-94
- refs golang/go#38789
- refs golang/go#26492
atc0005 added a commit to atc0005/check-mail that referenced this issue Oct 3, 2020
The previous build command produces dynamic executables
instead of the intended (but not well noted in the Makefile)
static executables. This commit includes changes noted in
upstream `golang/go` issues which appear to work as intended
for others.

- refs GH-94
- refs golang/go#38789
- refs golang/go#26492
@mark-rushakoff
Copy link
Contributor

@mark-rushakoff mark-rushakoff commented Mar 12, 2021

Since nobody has mentioned GOFLAGS yet...

I was updating our script that effectively runs:

readonly GO_LDFLAGS='-linkmode=external -extldflags "-fno-PIC -static -Wl,-z,stack-size=8388608" -buildid='
readonly GO_STATICTAGS='osusergo netgo static_build'

go build -ldflags "${GO_LDFLAGS}" -buildmode pie -tags "${GO_STATICTAGS}" "$@"

because it was hardcoded to go build and now we need to use those flags for go test -c.

I thought I would be smart and just pack everything into GOFLAGS so that the script condenses to roughly

export GOFLAGS=...
go "$@"

and callers could use static-go.sh build ./my/pkg or static-go.sh test -c ./my/pkg. But then I ran into #26849, and I could not get around the GOFLAGS parsing errors because the -ldflags arguments are space-separated.

Being able to specify GOFLAGS=-static would have saved me a few hours today.

@Scratch-net
Copy link

@Scratch-net Scratch-net commented Mar 23, 2021

How does -fno-PIC work with -buildmode pie ? Shouldn't it be -fPIC instead?

@noerw
Copy link

@noerw noerw commented May 10, 2021

To get a statically linked PIE executable I needed the following invocation (I'm mentioning this here, as I found this documented nowhere on the net, scraped it together from this thread, and this stackoverflow answer)

CGO_ENABLED=1 go build -buildmode=pie -tags 'osusergo,netgo,static,' -ldflags '-linkmode=external -extldflags "-static-pie"' .

The proposed -static flag would need to handle this case, when -buildmode=pie is also present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet