cmd/covdata: too many open files due to `defer f.Close()` in for loop

### Go version

go version go1.22.5 darwin/arm64

### Output of `go env` in your module/workspace:

```shell
GO111MODULE='auto'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/romain.marcadier/Library/Caches/go-build'
GOENV='/Users/romain.marcadier/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/romain.marcadier/go/pkg/mod'
GONOPROXY='github.com/DataDog'
GONOSUMDB='github.com/DataDog,go.ddbuild.io'
GOOS='darwin'
GOPATH='/Users/romain.marcadier/go'
GOPRIVATE='github.com/DataDog'
GOPROXY='binaries.ddbuild.io,proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD=''
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/qh/q2mpbd0j6xsb7vc8dqzdm81h0000gn/T/go-build2570734933=/tmp/go-build -gno-record-gcc-switches -fno-common'
```


### What did you do?

Trying to use `go tool covdata textfmt -i <some-dir> -o <some-report.txt>` results in errors such as:

```
error: error: opening counter data file coverage/raw/covcounters.848fe7b4490389df1276a71be50dd0d9.79237.1721148957056415000: open coverage/raw/covcounters.848fe7b4490389df1276a71be50dd0d9.79237.1721148957056415000: too many open files
```

This is the case when trying to consolidate coverage reporting from thousands of individual reports, which my use-case generates.

### What did you see happen?

The errors above, which happen on all go releases in our test suite so far:
- `go1.21`
- `go1.22`
- `go1.23` release candidate

### What did you expect to see?

I expect the tool to produce a report without needing to retain all files open at once.

---

Without having validated this claim (yet), I am pretty sure the culprit is this code:

https://github.com/golang/go/blob/8b48290895d362417124bcc18f0e6b6476ddc99e/src/cmd/internal/cov/readcovdata.go#L208-L239

Where the files are open in a loop that includes a `defer` statement to close the file; but the `defer` only happens at function exit, and the loop might go over a large number of files.

I would suggest refactoring the loop body to a new function/method as a way to reduce the lifetime of the open file descriptor here.

I might be able to submit a change request to gerrit (but not today).

	for k, cdf := range p.CounterDataFiles {
	cf, err := os.Open(cdf)
	if err != nil {
	return r.fatal("opening counter data file %s: %s", cdf, err)
	}
	defer func(f *os.File) {
	f.Close()
	}(cf)
	var mr *MReader
	mr, err = NewMreader(cf)
	if err != nil {
	return r.fatal("creating reader for counter data file %s: %s", cdf, err)
	}
	var cdr *decodecounter.CounterDataReader
	cdr, err = decodecounter.NewCounterDataReader(cdf, mr)
	if err != nil {
	return r.fatal("reading counter data file %s: %s", cdf, err)
	}
	r.vis.BeginCounterDataFile(cdf, cdr, p.Origins[k])
	var data decodecounter.FuncPayload
	for {
	ok, err := cdr.NextFunc(&data)
	if err != nil {
	return r.fatal("reading counter data file %s: %v", cdf, err)
	}
	if !ok {
	break
	}
	r.vis.VisitFuncCounterData(data)
	}
	r.vis.EndCounterDataFile(cdf, cdr, p.Origins[k])
	}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cmd/covdata: too many open files due to `defer f.Close()` in for loop #68468

Go version

Output of `go env` in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cmd/covdata: too many open files due to defer f.Close() in for loop #68468

Description

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

cmd/covdata: too many open files due to `defer f.Close()` in for loop #68468

Output of `go env` in your module/workspace: