os: File.Readdirnames order is different than of "ls -l"

[dxps]

What version of Go are you using (go version)?

$ go version
go version go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

Yes, this is the latest release at the time of this writing.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/devisions/.cache/go-build"
GOENV="/home/devisions/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/devisions/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/devisions/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/devisions/apps/go-1.15.6"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/devisions/apps/go-1.15.6/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build624368339=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Since I want to find a more performant alternative to ioutil.ReadDir (that takes around 1.2 seconds if there are over 100.000 files in the directory), this os.File.Readdirnames seems to be the perfect option.

But the order of entries that are provided does not seem to be consistent the same as of ls -l. Of course, some ordering can be done on the result, but the whole point was to quickly get the next couple of files from a directory, sorted either by name or creation time.

The API documentation mentions that:

Readdirnames reads the contents of the directory associated with file and returns a slice of up to n names of files in the directory, in directory order.

Now, what in directory order means is yet unclear to me.

The code used is extremely simple:

f, err := os.Open(path)
defer func() { _ = f.Close() }()
if err != nil {
	return "", err
}
fns, err := f.Readdirnames(100)
if err != nil {
	return "", err
}
log.Printf("[dbg] Readdirnames => %v\n", fns)

What did you expect to see?

I was expected to see the same output as of ls:

1609349600755907389.dat  1609349614877620779.dat  1609349614878350488.dat  1609349615380619680.dat

What did you see instead?

But - a concrete output, related to the same content as previously listed - the result of running the code is:

[dbg] Readdirnames => [1609349615380619680.dat 1609349600755907389.dat 1609349614877620779.dat 1609349614878350488.dat]

Any advice, please?

Thank you!

[dmitshur]

Directory order means the same order that the operating system returns. This depends on the OS type, version, and importantly the filesystem used at the path that was read.

To get a predictable order across systems, you need to perform sorting yourself, as done by ioutil.ReadDir.

I think this is working as intended and documented. Please comment if I missed something.

[dxps]

@dmitshur Thanks for the feedback.

brb with the result.

[dmitshur]

I'll close this as there's no bug here, and we use the Go issue tracker for bugs and proposals only.

For questions and discussion about using Go, please see https://golang.org/wiki/Questions for a list of better places. Thanks.

P.S. In your snippet, please note that you should handle the error from os.Open first, and call defer f.Close() second, not vice versa. Otherwise, the code may call Close on an invalid file.

[dxps]

As promised, got back with the result. I was about to close this issue anyway... 😊

As a summary, I am happy with these figures:

• although sorting is needed (basically, used fnames, err = f.Readdirnames(0) then sort.Strings(fnames))
• the overall spent time is considerable smaller than of ioutil.ReadDir's one.

>>> ioutil.ReadDir exec time: 1.311758808s
>>> os.File.Readdirnames exec time: 144.131977ms
>>> os.File.Readdirnames result has 499099 entries.
>>> sort exec time: 163.322515ms

P.S. You're right, thanks! The code was just for this sample purposes.

[dmitshur]

Yep, the time difference is expected because ioutil.ReadDir calls f.Readdir which performs lstat for each file to get more information (os.FileInfo), while f.Readdirnames(0) returns only the file names.

Note that starting with Go 1.16, there will be a new API os.ReadDir that you might find more convenient to use. It's fast by default, but lets you get more information on demand.

[dxps]

Yes, read about the upcoming API. Thanks.