testing/fstest: fstest.TestFS is slow when given directory contains relatively big files #69888
Description
Go version
go version go1.23.2 darwin/arm64
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/asulim/Library/Caches/go-build'
GOENV='/Users/asulim/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/packages/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/packages'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/asulim/.local/share/asdf/installs/golang/1.23.2/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/asulim/Library/Application Support/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/asulim/data/sources/github.com/soulim/testfstest/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/wf/vzjbb9654pd88ndz56k77xqh0000gn/T/go-build734442337=/tmp/go-build -gno-record-gcc-switches -fno-common'What did you do?
A call to fstest.TestFS takes significant time (seconds) to check a directory, when it contains at least one relatively big file.
I've created a demo repository with a set of tests relying on fstest.TestFS: https://github.com/soulim/testfstest. Additionally there's a GitHub workflow that runs the same tests to demonstrate results.
Setup
testdata/many-bigdirectory contains a set of files and one of them is big (4.4MB).testdata/many-smalldirectory contains a set of file, but none of them is big.testdata/one-<category>directories contain only one file of each category: big, small, and empty.
Each time when a test works with a directory containing a big file, it takes few seconds for fstest.TestFS to return results.
What did you see happen?
An example test with fstest.TestFS:
func TestTestFS(t *testing.T) {
files := []string{
filepath.Join("testdata", "many-big", "big.jpg"),
filepath.Join("testdata", "many-big", "empty.txt"),
filepath.Join("testdata", "many-big", "small.txt"),
filepath.Join("testdata", "many-small", "empty.txt"),
filepath.Join("testdata", "many-small", "small.txt"),
filepath.Join("testdata", "one-big", "big.jpg"),
filepath.Join("testdata", "one-empty", "empty.txt"),
filepath.Join("testdata", "one-small", "small.txt"),
}
for _, file := range files {
t.Run(file, func(t *testing.T) {
var ts0, ts1 time.Time
dir := filepath.Dir(file)
ts0 = time.Now()
err := fstest.TestFS(os.DirFS(dir), filepath.Base(file))
ts1 = time.Now()
t.Logf("TestFS() time: %v", ts1.Sub(ts0))
if err != nil {
t.Fatal(err)
}
})
}
}Output:
go test ./... -v
=== RUN TestTestFS
=== RUN TestTestFS/testdata/many-big/big.jpg
main_test.go:32: TestFS() time: 4.008915917s
=== RUN TestTestFS/testdata/many-big/empty.txt
main_test.go:32: TestFS() time: 3.955715375s
=== RUN TestTestFS/testdata/many-big/small.txt
main_test.go:32: TestFS() time: 3.953422583s
=== RUN TestTestFS/testdata/many-small/empty.txt
main_test.go:32: TestFS() time: 2.408916ms
=== RUN TestTestFS/testdata/many-small/small.txt
main_test.go:32: TestFS() time: 721.459µs
=== RUN TestTestFS/testdata/one-big/big.jpg
main_test.go:32: TestFS() time: 3.968942875s
=== RUN TestTestFS/testdata/one-empty/empty.txt
main_test.go:32: TestFS() time: 243.958µs
=== RUN TestTestFS/testdata/one-small/small.txt
main_test.go:32: TestFS() time: 2.164458ms
--- PASS: TestTestFS (15.89s)
--- PASS: TestTestFS/testdata/many-big/big.jpg (4.01s)
--- PASS: TestTestFS/testdata/many-big/empty.txt (3.96s)
--- PASS: TestTestFS/testdata/many-big/small.txt (3.95s)
--- PASS: TestTestFS/testdata/many-small/empty.txt (0.00s)
--- PASS: TestTestFS/testdata/many-small/small.txt (0.00s)
--- PASS: TestTestFS/testdata/one-big/big.jpg (3.97s)
--- PASS: TestTestFS/testdata/one-empty/empty.txt (0.00s)
--- PASS: TestTestFS/testdata/one-small/small.txt (0.00s)
PASS
ok github.com/soulim/testfstest 16.170sAs you could see above it took 15 seconds for the test suite to finish. Each test is only using fstest.TestFS.
What did you expect to see?
I did check the documentation for fstest.TestFS and haven't found any warning regarding working with big files.
fstest.TestFS does many checks and maybe it's expected for it to be slow with big files. However in that case, it would be nice to inform about that in the documentation.
I'm more than happy to provide any additional details.
Thank you a lot for being awesome!