New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: large arrays cause large binaries and toolchain errors #17378

Open
crawshaw opened this Issue Oct 7, 2016 · 6 comments

Comments

Projects
None yet
9 participants
@crawshaw
Contributor

crawshaw commented Oct 7, 2016

On tip, go version devel +0db9518 Fri Oct 7 03:46:50 2016 +0000 darwin/amd64:

$ cat j.go 
package main

var x = [1 << 30]byte{0: 1}

func main() {
    println(x[0])
}
$ go build j.go
$ ls -l j
-rwxr-xr-x  1 crawshaw  eng  1074669392 Oct  7 10:34 j

Bumping 1<<30 to 1<<31 turns it into a linker error:

~$ go build j.go 
# command-line-arguments
main.x: symbol too large (2147483648 bytes)
too much data in section SNOPTRDATA (over 2000000000 bytes)
too much data in section SDATA (over 2000000000 bytes)
too much data in section SBSS (over 2000000000 bytes)
too much data in section SNOPTRBSS (over 2000000000 bytes)
main.init: pc-relative relocation address for main.initdone· is too big: 0x800ad36a
main.init: pc-relative relocation address for main.initdone· is too big: 0x800ad355
runtime.alginit: pc-relative relocation address for runtime.cpuid_ecx is too big: 0x800ac585
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac72d
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac70b
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac700
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac701
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6f6
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6f7
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6ec
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6ed
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6e2
runtime.alginit: pc-relative relocation address for runtime.useAeshash is too big: 0x800ac4ea
runtime.alginit: pc-relative relocation address for runtime.algarray is too big: 0x80091a0b
runtime.alginit: pc-relative relocation address for runtime.algarray is too big: 0x80091a0d
runtime.alginit: pc-relative relocation address for runtime.algarray is too big: 0x80091a1f
/Users/crawshaw/go/pkg/tool/darwin_amd64/link: too many errors

(Inspired by https://codegolf.stackexchange.com/questions/69189/build-a-compiler-bomb/69193#69193)

@mrjrieke

This comment has been minimized.

mrjrieke commented Oct 7, 2016

I wonder how compiler behaves if allocate is inside main?

@ALTree

This comment has been minimized.

Member

ALTree commented Oct 7, 2016

Isn't this on purpose? The message was introduced in commit cmd/link: reject data size > 2 GB to fix issue #9862

@mrjrieke

This comment has been minimized.

mrjrieke commented Oct 7, 2016

True Alberto. Although, does golint help identify the line number with the
problem? Seems like something either the compiler or one of the tools
should be able to identify more precisely the source of the problem.

On Fri, Oct 7, 2016 at 9:52 AM, Alberto Donizetti notifications@github.com
wrote:

Isn't this on purpose? The message was introduced in commit cmd/link:
reject data size > 2 GB to fix issue #9862
#9862


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#17378 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATlFP4yqQAQDLDsBfd-FPyaigmcc4QzUks5qxnjrgaJpZM4KRGlQ
.

@quentinmit quentinmit added this to the Go1.8 milestone Oct 9, 2016

@rsc rsc modified the milestones: Go1.9Early, Go1.8 Oct 21, 2016

@bradfitz bradfitz modified the milestones: Go1.9Maybe, Go1.9Early May 3, 2017

@bradfitz bradfitz modified the milestones: Go1.9Maybe, Go1.10 Jul 20, 2017

@mdempsky mdempsky modified the milestones: Go1.10, Go1.11 Nov 30, 2017

@mdempsky

This comment has been minimized.

Member

mdempsky commented Nov 30, 2017

It's not obvious to me that this issue is actionable.

@marigonzes

This comment has been minimized.

marigonzes commented Apr 6, 2018

I encountered this issue too.
The simplest case I could come up with was https://play.golang.org/p/EK8zDu5vVQN (creates a ~800MB executable).
The thing I find weird is that https://play.golang.org/p/LM9TBaQcRIo works just fine and their assembly code isn't that different (when compiling with 'go tool compile -S ...', because after using objdump the code is totally different).

@marigonzes

This comment has been minimized.

marigonzes commented Apr 7, 2018

I think I found the value where the first jump in the size of the executable occurs:

  • When I use an array size of 1310720, the executable is ~1MB;
  • When I change it to 1310721, the executable is ~11MB.

Hope this helps debug the problem.

@gopherbot gopherbot modified the milestones: Go1.11, Unplanned May 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment