proposal: os: make Readdir return lazy FileInfo implementations #41188
Description
An os.File provides two ways to read a directory: Readdirnames returns a list of the names of the directory entries, and Readdir returns the names along with stat information.
On Plan 9 and Windows, Readdir can be implemented with only a directory read - the directory read operation provides the full stat information.
But many Go users use Unix systems.
On most Unix systems, the directory read does not provide full stat information. So the implementation of Readdir reads the names from the directory and then calls Lstat for each file. This is fairly expensive.
Much of the time, such as in the implementation of filepath.Glob and other file system walking, the only information the caller of Readdir really needs is the name and whether the name denotes a directory. On most Unix systems, that single bit of information—is this name a directory?—is available from the plain directory read, without an additional stat. If the caller is only using that bit, the extra Lstat calls are unnecessary and slow. (Goimports, for example, has its own directory walker to avoid this cost.)
Various people have proposed adding a third directory reading option of one form or another, to get names and IsDir bits. This would certainly address the slow directory walk issue on Unix systems, but it seems like overfitting to Unix.
Note that os.FileInfo is an interface. What if we make Readdir return a slice of lazily-filled os.FileInfo? That is, on Unix, Readdir would stop calling Lstat. Each returned FileInfo would already know the answer for its Name and IsDir methods. The first call to any of the other methods would incur an Lstat at that moment to find out the rest of the information. A directory walk that uses Readdir and then only calls Name and IsDir would have all its Lstat calls optimized away with no code changes in the caller.
The downside of this is that the laziness would be visible when you do the Readdir and wait a while before looking at the results. For example if you did Readdir, then touched one of the files in the list, then called the ModTime method on the os.FileInfo that Readdir retruned, you'd see the updated modification time. And then if you touched the file again and called ModTime again, you wouldn't see the further-updated modification time. That could be confusing. But I expect that the vast majority of uses of Readdir use the results immediately or at least before making changes to files listed in the results. I suspect the vast majority of users would not notice this change.
I propose we make this change—make Readdir return lazy os.FileInfo—soon, intending it to land in Go 1.16, but ready to roll back the change if the remainder of the Go 1.16 dev cycle or beta/rc testing turns up important problems with it.