Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: linking github.com/hashicorp/packer takes >2gb of ram #32094

Closed
lyda opened this issue May 17, 2019 · 9 comments
Closed

cmd/link: linking github.com/hashicorp/packer takes >2gb of ram #32094

lyda opened this issue May 17, 2019 · 9 comments
Milestone

Comments

@lyda
Copy link

@lyda lyda commented May 17, 2019

What version of Go are you using (go version)?

Using the version from ports:

$ go version
go version go1.12.5 freebsd/amd64

Does this issue reproduce with the latest release?

I think that's the latest release, so yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOOS="freebsd"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build861311780=/tmp/go-build -gno-record-gcc-switches"

What did you do?

As mentioned in #27739, building packer on a t2.small with around 1.5g of ram will fail. A full terraform example is here and will reproduce the problem.

What did you expect to see?

I expected packer to be linked.

What did you see instead?

The Go compiler crashed at the link stage. If you follow the terraform example and upgrade to t2.medium, you'll see it finally succeed - it will have 3.8g of ram available (check with vmstat).

@odeke-em odeke-em changed the title Go compiler runs out of memory on t2.small when building packer cmd/compile: out of memory on t2.small when building packer with Terraform May 17, 2019
@odeke-em odeke-em changed the title cmd/compile: out of memory on t2.small when building packer with Terraform cmd/compile: out of memory on t2.small when linking/building packer with Terraform May 17, 2019
@odeke-em
Copy link
Member

@odeke-em odeke-em commented May 17, 2019

Hello @lyda thank you for reporting this issue and welcome to the Go project!

I'll page some experts about this @ianlancetaylor @randall77 @josharian @aclements, but am not sure if we should triage this for Go1.13 or Go1.14?

@lyda
Copy link
Author

@lyda lyda commented May 17, 2019

Thanks. And no worries about versions, whichever version makes sense. I was using the one users of FreeBSD are likely to use.

Note, I provided a simple terraform config to make it easier to reproduce. Finding the right FreeBSD AMI and navigating AWS if you don't know it is a pain. So the terraform config is just there to make it easier for you to get a test machine to try it on.

You can get the same outcome on a similarly sized non-AWS VM, I just happened to be setting up an AWS instance at the time. The base issue is memory usage in linking. The packer project seems to present an example body of code that takes what seems like an excessive amount of ram to link.

I think the changed title is a bit misleading because the issue really isn't with terraform. It's merely a convenience I provided to make people's lives easier in reproducing the issue. And I only included packer in the title because that's what I was compiling - I suspect packer is hardly the only code base where this issue occurs but I have not tried compiling a bunch of Go projects to prove that.

@odeke-em odeke-em changed the title cmd/compile: out of memory on t2.small when linking/building packer with Terraform cmd/compile: out of memory on t2.small when linking/building with less than 1.5GB of RAM May 17, 2019
@odeke-em
Copy link
Member

@odeke-em odeke-em commented May 17, 2019

Thank you @lyda! I've updated it to "cmd/compile: out of memory on t2.small when linking/building with less than 1.5GB of RAM", please feel free to modify it. You have access too to update the title :)

@josharian
Copy link
Contributor

@josharian josharian commented May 17, 2019

I tried doing GO111MODULE=off /usr/bin/time -l go build -a github.com/hashicorp/packer with Go 1.11, 1.12, and tip. The maxrss was 2.25gb, 2.49gb, and 2.09gb respectively. So 1.13 is moving in the right direction.

cc @thanm and @cherrymui who have been working on linker memory reductions

I'm going to tentatively mark this as 1.13, in case either of them sees something easy and safe to do here, inspired particularly by linking packer. It is highly likely this will get punted to 1.14, though.

@josharian josharian added this to the Go1.13 milestone May 17, 2019
@josharian josharian changed the title cmd/compile: out of memory on t2.small when linking/building with less than 1.5GB of RAM cmd/link: linking github.com/hashicorp/packer takes >2gb of ram May 17, 2019
@lyda
Copy link
Author

@lyda lyda commented May 17, 2019

2.13 looks promising - just need to shave off .6g or so! :)

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented May 17, 2019

Quick question: Does it use cgo (i.e. internal linking or external linking)? Do you build a static executable, or PIE/shared library/etc. which requires dynamic relocations? Also, how big are the inputs (object files) and the output? Thanks!

@lyda
Copy link
Author

@lyda lyda commented May 17, 2019

Not really sure.

# file /usr/local/bin/packer
/usr/local/bin/packer: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, Go BuildID=F_YMeGLIsqFoRNLB81mG/2EtlqnuY6GqeXjDDVPun/WiejIm7lSAh-xMMbUR98/kcs28g1e9gfaaebfZlUt, stripped

It's made with this Makefile. It includes a lot, but I think go.mk is the relevant one.

@ALTree
Copy link
Member

@ALTree ALTree commented Jul 21, 2020

It looks like the recent linker improvements fixed this issue.

I tried building github.com/hashicorp/packer with tip (soon to be go1.15) on an aws t2.small instance, and it worked.

$ ec2-metadata -t
instance-type: t2.small

$ grep MemTotal /proc/meminfo
MemTotal:        2039152 kB

$ ~/gotip/bin/go version
go version devel +11f92e9 Mon Jul 20 18:18:56 2020 +0000 linux/amd64

$ /usr/bin/time -v ~/gotip/bin/go build -a github.com/hashicorp/packer
	User time (seconds): 309.87
	System time (seconds): 33.28
	Percent of CPU this job got: 95%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 5:58.23
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1757880
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 4988
	Minor (reclaiming a frame) page faults: 8098532
	Voluntary context switches: 242554
	Involuntary context switches: 162508
	Swaps: 0
	File system inputs: 2104784
	File system outputs: 2374984
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

I'm closing this issue, but feel free to comment if you believe it should be kept open.

@ALTree ALTree closed this Jul 21, 2020
@aclements
Copy link
Member

@aclements aclements commented Jul 22, 2020

Thanks for closing the loop on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.