os: on Unix, Readdirnames skips directory entries with zero inodes

[vasi]

Go version

go version go1.25.4 X:nodwarf5 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/vasi/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/vasi/.config/go/env'
GOEXE=''
GOEXPERIMENT='nodwarf5'
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1615998593=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/vasi/src/restic/go.mod'
GOMODCACHE='/home/vasi/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/vasi/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/vasi/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.4 X:nodwarf5'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

1. Mount a filesystem that returns direntries with zero inodes. On a Linux system with libfuse 3.17.1 or later, you can do this easily with GVFS:

   # Create many dir entries
   (cd big && printf '%04x ' {0..1023} | xargs mkdir -p)
   gio mount sftp://localhost/$PWD/big
   # Verify that some entries have zero inodes:
   python -c 'import os; import sys; print([d.inode() for d in os.scandir(sys.argv[1])])' /run/user/$UID/gvfs/sftp:host=localhost/$PWD/big
   

2. List the directory with a tool that uses glibc 2.37 or later. Eg: ls -a /run/user/$UID/gvfs/sftp:host=localhost/$PWD/big | wc -l. It returns all 1026 entries (including . and ..).
3. List the directory with Go:

   package main
   
   import (
   	"fmt"
   	"os"
   )
   
   func main() {
   				dir := os.Args[1]
   	f, err := os.Open(dir)
   	if err != nil {
   		panic(err)
   	}
   	defer f.Close()
   
   	names, err := f.Readdirnames(-1)
   	if err != nil {
   		panic(err)
   	}
   	fmt.Printf("Readdirnames: %d items\n", len(names))
   }
   

   Run: go run ./main.go /run/user/$UID/gvfs/sftp:host=localhost/$PWD/big

What did you see happen?

Go prints: Readdirnames: 49 items, ignoring the majority of the directory entries.

What did you expect to see?

Go sees all 1024 directory entries.

[gabyhelp]

Related Issues

• os: ReadDir on regular file returns non nil slice on Windows #75157 (closed)
• os: document that fs.ReadDirFile.ReadDir doesn't restart each time #69301 (closed)
• syscall: incorrect length of abstract Unix domain socket paths beginning with null character #63579 (closed)
• os: reading the contents of a directory can result in "lost" files #24015 (closed)

Related Code Changes

• os: do not skip directory entries with zero inodes on wasip1

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[vasi]

Some more context on this:

• The bug is triggered here in dir_unix.go, as well as here in syscall/dirent.go
• Skipping an entry with zero d_ino may be legal in POSIX, though I'm having trouble finding any specific text around it. Here's a claim on linux-fsdevel that a zero d_ino means the file has been removed. Nevertheless, since 2022 glibc does not skip these entries, nor do most Unix tools aside from Go, and Go should aim for consistency with other tools.
• libfuse returning zero d_ino is itself probably a bug, but several versions of libfuse with this bug are in the wild, so we should probably be able to deal with them. There's also other filesystems that may return zero d_ino, for example the glibc change references old versions of XFS doing this.
• This bug is the cause of a bug in Restic and therefore in Deja Dup as well.

[ianlancetaylor]

POSIX doesn't say anything about a zero d_ino, but I don't think that's relevant because POSIX is talking about the readdir_r API and friends. Go is, essentially, implementing that API using lower-level calls. On Linux Go uses getdents. That returns a series of dirent structs.

Historically on a Unix system you could just read a directory as a regular file, and get a series of dirent (or direct) structs. In that model a deleted file would have a d_ino field of 0, and it was necessary to skip those entries. It sounds like, at least on Linux, things have changed. But it's not clear that they've changed on other operating systems. In particular I see that the OpenBSD getdents man page says "Invalid entries with d_fileno set to 0 may be returned among regular entries."

So for now I think we should change the code in (*os.File).readdir and syscall.ParseDirent to add "linux" to the existing "wasip1" exception for a 0 inode.

[cherrymui]

cc @rsc @bradfitz @griesemer

[mikix]

This line from the man page for posix_getdents feels relevant (and agrees with the above analysis - some kernels flag invalid entries, some don't, but POSIX itself has no special values here):

Some existing getdents() or similar functions return directory entry structures for deleted directory entries in buf, marked with a special value of one of the structure members to distinguish them from non-deleted entries. This behavior is not allowed for posix_getdents(), although the data from a deleted directory entry may be present in buf in the form of extra padding on the end of the previous entry.

[gopherbot]

Change https://go.dev/cl/724220 mentions this issue: os: allow direntries to have zero inodes on Linux