Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: binary on darwin takes up more space on disk #39044

Closed
kokes opened this issue May 13, 2020 · 7 comments
Closed

cmd/link: binary on darwin takes up more space on disk #39044

kokes opened this issue May 13, 2020 · 7 comments

Comments

@kokes
Copy link

@kokes kokes commented May 13, 2020

What version of Go are you using (go version)?

$ go version
go version devel +8ab37b1baf Mon Apr 20 18:32:58 2020 +0000 darwin/amd64

Does this issue reproduce with the latest release?

No (as in this is a regression in the current master, the latest stable is fine)

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/okokes/Library/Caches/go-build"
GOENV="/Users/okokes/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/okokes/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.2_1/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.2_1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/81/4jydp7kn51n6p68z88sqnkzc0000gn/T/go-build271357823=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I noticed my binary went from 8MB (1.14.2) to 13MB (master), but only when running du -sh or inspecting the binary in Finder (under "how much it takes on disk").

When stating both binaries, they are very similar in size, so this issue only revolves around disk usage, not size.

I replicated the issue by creating a hello world app

package main

import "fmt"

func main() {
fmt.Println("ahoy")
}

And then I bisected it, starting at 1811533 (good) and ending at cb11c98 (bad).

package main

import (
"log"
"os/exec"
)

func main() {
build := exec.Command("/Users/okokes/git/go/src/make.bash")
build.Dir = "/Users/okokes/git/go/src"
err := build.Run()
if err != nil {
log.Fatal("toolchain build failed", err)
}

cmd := exec.Command("/Users/okokes/git/go/bin/go", "build", "src/hello.go")
err = cmd.Run()
if err != nil {
	log.Fatal("program build failed", err)
}

sz := exec.Command("du", "-sh", "hello")
out, err := sz.Output()
if err != nil {
	log.Fatal("du failed", err)
}
num := out[0]
if num != '2' {
	log.Fatal(string(out))
}

}

Bisect identified 8ab37b1 as the first offending commit. I verified it manually - the commit before that leads to a 2.1MB binary on disk, this commit leads to 4.1MB on disk.

I could not replicate this on a Ubuntu 18.04 box, so I presume it's a Darwin thing.

What did you expect to see?

$ stat -f '%z %N' hello_*
2216280 hello_8ab37b1
2174008 hello_go1.14
$ du -sh hello_*
2.1M	hello_8ab37b1
2.1M	hello_go1.14

What did you see instead?

$ stat -f '%z %N' hello_*
2216280 hello_8ab37b1
2174008 hello_go1.14
$ du -sh hello_*
4.1M	hello_8ab37b1     <-- note the binary size here
2.1M	hello_go1.14
@bradfitz bradfitz added this to the Go1.16 milestone May 13, 2020
@bradfitz bradfitz added the OS-Darwin label May 13, 2020
@josharian
Copy link
Contributor

@josharian josharian commented May 14, 2020

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented May 18, 2020

@randall77
Copy link
Contributor

@randall77 randall77 commented May 18, 2020

Note that copying the result to a new file fixes the problem.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented May 18, 2020

Hmmm. https://tip.golang.org/src/cmd/link/internal/ld/outbuf_darwin.go

func (out *OutBuf) fallocate(size uint64) error {
	store := &syscall.Fstore_t{
		Flags:   syscall.F_ALLOCATEALL,
		Posmode: syscall.F_PEOFPOSMODE,
		Offset:  0,
		Length:  int64(size),
	}

	_, _, err := syscall.Syscall(syscall.SYS_FCNTL, uintptr(out.f.Fd()), syscall.F_PREALLOCATE, uintptr(unsafe.Pointer(store)))
	if err != 0 {
		return err
	}

	return nil
}

I wonder if F_PEOFPOSMODE is right here. It seems

F_PEOFPOSMODE Allocate from the physical end of file.

So if we preallocate more than once, the second and later calls will allocate the whole size on top of the existing size...

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented May 18, 2020

Or we should subtract the existing size.

@gopherbot
Copy link

@gopherbot gopherbot commented May 18, 2020

Change https://golang.org/cl/234481 mentions this issue: cmd/link: fix size calculation for file space preallocation on darwin

@networkimprov
Copy link

@networkimprov networkimprov commented May 18, 2020

Shouldn't this be milestoned for 1.15?

@gopherbot gopherbot closed this in 39ea0ea May 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.