Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: linking github.com/hashicorp/packer takes >2gb of ram #32094

Open
lyda opened this issue May 17, 2019 · 7 comments

Comments

@lyda
Copy link

commented May 17, 2019

What version of Go are you using (go version)?

Using the version from ports:

$ go version
go version go1.12.5 freebsd/amd64

Does this issue reproduce with the latest release?

I think that's the latest release, so yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOOS="freebsd"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build861311780=/tmp/go-build -gno-record-gcc-switches"

What did you do?

As mentioned in #27739, building packer on a t2.small with around 1.5g of ram will fail. A full terraform example is here and will reproduce the problem.

What did you expect to see?

I expected packer to be linked.

What did you see instead?

The Go compiler crashed at the link stage. If you follow the terraform example and upgrade to t2.medium, you'll see it finally succeed - it will have 3.8g of ram available (check with vmstat).

@odeke-em odeke-em changed the title Go compiler runs out of memory on t2.small when building packer cmd/compile: out of memory on t2.small when building packer with Terraform May 17, 2019
@odeke-em odeke-em changed the title cmd/compile: out of memory on t2.small when building packer with Terraform cmd/compile: out of memory on t2.small when linking/building packer with Terraform May 17, 2019
@odeke-em

This comment has been minimized.

Copy link
Member

commented May 17, 2019

Hello @lyda thank you for reporting this issue and welcome to the Go project!

I'll page some experts about this @ianlancetaylor @randall77 @josharian @aclements, but am not sure if we should triage this for Go1.13 or Go1.14?

@lyda

This comment has been minimized.

Copy link
Author

commented May 17, 2019

Thanks. And no worries about versions, whichever version makes sense. I was using the one users of FreeBSD are likely to use.

Note, I provided a simple terraform config to make it easier to reproduce. Finding the right FreeBSD AMI and navigating AWS if you don't know it is a pain. So the terraform config is just there to make it easier for you to get a test machine to try it on.

You can get the same outcome on a similarly sized non-AWS VM, I just happened to be setting up an AWS instance at the time. The base issue is memory usage in linking. The packer project seems to present an example body of code that takes what seems like an excessive amount of ram to link.

I think the changed title is a bit misleading because the issue really isn't with terraform. It's merely a convenience I provided to make people's lives easier in reproducing the issue. And I only included packer in the title because that's what I was compiling - I suspect packer is hardly the only code base where this issue occurs but I have not tried compiling a bunch of Go projects to prove that.

@odeke-em odeke-em changed the title cmd/compile: out of memory on t2.small when linking/building packer with Terraform cmd/compile: out of memory on t2.small when linking/building with less than 1.5GB of RAM May 17, 2019
@odeke-em

This comment has been minimized.

Copy link
Member

commented May 17, 2019

Thank you @lyda! I've updated it to "cmd/compile: out of memory on t2.small when linking/building with less than 1.5GB of RAM", please feel free to modify it. You have access too to update the title :)

@josharian

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

I tried doing GO111MODULE=off /usr/bin/time -l go build -a github.com/hashicorp/packer with Go 1.11, 1.12, and tip. The maxrss was 2.25gb, 2.49gb, and 2.09gb respectively. So 1.13 is moving in the right direction.

cc @thanm and @cherrymui who have been working on linker memory reductions

I'm going to tentatively mark this as 1.13, in case either of them sees something easy and safe to do here, inspired particularly by linking packer. It is highly likely this will get punted to 1.14, though.

@josharian josharian added this to the Go1.13 milestone May 17, 2019
@josharian josharian changed the title cmd/compile: out of memory on t2.small when linking/building with less than 1.5GB of RAM cmd/link: linking github.com/hashicorp/packer takes >2gb of ram May 17, 2019
@lyda

This comment has been minimized.

Copy link
Author

commented May 17, 2019

2.13 looks promising - just need to shave off .6g or so! :)

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

Quick question: Does it use cgo (i.e. internal linking or external linking)? Do you build a static executable, or PIE/shared library/etc. which requires dynamic relocations? Also, how big are the inputs (object files) and the output? Thanks!

@lyda

This comment has been minimized.

Copy link
Author

commented May 17, 2019

Not really sure.

# file /usr/local/bin/packer
/usr/local/bin/packer: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, Go BuildID=F_YMeGLIsqFoRNLB81mG/2EtlqnuY6GqeXjDDVPun/WiejIm7lSAh-xMMbUR98/kcs28g1e9gfaaebfZlUt, stripped

It's made with this Makefile. It includes a lot, but I think go.mk is the relevant one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.