cmd/link: too many open files on high object-count dependencies

[karalabe]

I think this issue is a continuation of #21621, just in a later step when using a huge package, not when building it.

What version of Go are you using (go version)?

$ go version
go version go1.11.2 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/karalabe/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/work"
GOPROXY=""
GORACE=""
GOROOT="/opt/google/go"
GOTMPDIR=""
GOTOOLDIR="/opt/google/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build076342095=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I've built a Go library consisting of more than 1024 Go files (wrapper for Tor): github.com/ipsn/go-libtor. I can successfully build this library into a package (note, only Linux and Android is supported for now).

When using this package in another project however, Go's linker uses as many file descriptors as the number of object files the package has. Exceeding the 1024 object file limit, Go cannot link against this package any more on Linux without raising the ulimit.

What did you expect to see?

Successful compilation.

What did you see instead?

Link fails with too many open files.

Reproduction

I figured it's easier to show-and-tell. First up, prepare an empty GOPATH so we don't mess with anything else:

$ mkdir -p /tmp/gopath
$ cd /tmp/gopath/
$ export GOPATH=/tmp/gopath/

Download and build the Tor wrapper, this takes about 3 minutes on my laptop (verbosity added so you can track the progress, there are 919 files (yes, this is not the case that exceeds 1024 because I didn't want to break the lib, but we'll adjust ulimit to repro nonetheless)):

$ go get -x -v github.com/ipsn/go-libtor

Ok, lets create a tiny repro program that just imports the library we packaged up previously. Create a small package and dump the contents into it:

$ mkdir -p /tmp/gopath/src/repro
$ cd /tmp/gopath/src/repro
$ cat main.go
package main

import (
	"os"

	"github.com/cretz/bine/tor"
	"github.com/ipsn/go-libtor"
)

func main() {
	t, err := tor.Start(nil, &tor.StartConf{ProcessCreator: libtor.Creator, DebugWriter: os.Stderr})
	if err != nil {
		panic(err)
	}
	t.Close()
}

Fetch any missing dependencies and run it, you should see some logs from Tor:

$ go get -v -x
$ go run main.go 
[... logs from Tor ...]

Now lower the open file descriptors to below the library file count of 919 and try again:

$ ulimit -n 512
$ go run main.go 
# command-line-arguments
/opt/google/go/pkg/tool/linux_amd64/link: cannot create /tmp/go-link-924865611/000536.o: open /tmp/go-link-924865611/000536.o: too many open files
/opt/google/go/pkg/tool/linux_amd64/link: cannot create /tmp/go-link-924865611/000540.o: open /tmp/go-link-924865611/000540.o: too many open files
/opt/google/go/pkg/tool/linux_amd64/link: cannot create /tmp/go-link-924865611/000538.o: open /tmp/go-link-924865611/000538.o: too many open files
/opt/google/go/pkg/tool/linux_amd64/link: cannot create /tmp/go-link-924865611/000539.o: open /tmp/go-link-924865611/000539.o: too many open files
/opt/google/go/pkg/tool/linux_amd64/link: cannot reopen /tmp/gopath/pkg/linux_amd64/github.com/ipsn/go-libtor.a(_x470.o): open /tmp/gopath/pkg/linux_amd64/github.com/ipsn/go-libtor.a: too many open files
/opt/google/go/pkg/tool/linux_amd64/link: cannot create /tmp/go-link-924865611/000470.o: open /tmp/go-link-924865611/000470.o: too many open files

[ianlancetaylor]

I see the problem, sending a CL.

[gopherbot]

Change https://golang.org/cl/152757 mentions this issue: cmd/link: close input files when copying to temporary directory

[bradfitz]

@gopherbot, please backport to Go 1.11.

[gopherbot]

Backport issue(s) opened: #29112 (for 1.11).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/152760 mentions this issue: [release-branch.go1.11] cmd/link: close input files when copying to temporary directory

[karalabe]

Can confirm this fix solved the issue. Thank you!