proposal: os: new Readdirentries method to read directory entries and efficiently expose file metadata

[israel-lugo]

API

(Update: edited to turn into a proposal, add suggested API)

func (f *File) Readdirentries(n int) ([]DirEntry, error)

// The file type could not be determined
const ModeUnknown FileMode = xxx

type DirEntry struct {
  // Name is the base name of the file
  Name string
  // ... unexported fields ...
}

// Id returns a number which uniquely identifies the file within the filesystem that contains it.
// This is known as the inode number in Unix-like systems, or file ID in Windows. Under Windows,
// this will require a system call on first call, but not on Unix.
//
// Note this is not guaranteed to be unique in the ReFS file system, introduced with Windows
// Server 2012, since that uses 128-bit identifiers.
func (d *DirEntry) Id() (uint64, error) { ... }

// Type returns the file's type. Depending on the underlying filesystem, this may require an Lstat,
// which will be done internally and cached after first use. If lightweight is true, the Lstat will not
// be done; in that case, if the file type is not immediately known, ModeUnknown will be
// returned. This may be useful e.g.if the caller will be opening the file anyway and would prefer
// to do a Stat of the open file to avoid filename races. 
func (d *DirEntry) Type(lightweight bool) (FileMode, error) { ... }

// Lstat behaves like the normal Lstat. Its result will be cached after the first use, which may have
// occurred from calling Type or even Inode under Windows.
func (d *DirEntry) Lstat() (FileInfo, error) { ... }

This is analogous e.g. to Python's os.scandir, as pointed out by @qingyunha below.

Context

Could we please have a new File-level API to list a directory's entries, which exposes the d_type field (syscall.Dirent.Type) and, ideally, also d_ino (syscall.Dirent.Ino)?

Under Linux, certain filesystems (such as Btrfs, ext4) store the file type information in the direntry itself. This is available via the d_type field, which may be DT_UNKNOWN if the file type could not be determined for some reason (e.g. no filesystem support, or weird quirks such as "." or ".."). According to man readdir(3), some BSDs also support this.

Currently, we have os.(*File).Readdir, which does an lstat on every file and does not make use of the type information, even if it's there. This makes sense given the method's signature, since it needs to find out the file's size, mode, etc.

We also have os.(*File).Readdirnames, which reads the dirent but only returns the name portion.

It would be very useful to have an intermediate method between these two, that returns not only the name, but also the file type (which may of course be DT_UNKNOWN), and ideally anything else it can know from the dirent, such as the file's inode number (d_ino or syscall.Dirent.Ino).

This would make it much easier to implement a fast/scalable file crawler (e.g. for backup software or something else). Given a directory with 100,000 entries, being able to cheaply separate subdirectories from other files while listing the directory itself lets the crawler e.g. batch up regular files for further processing, or choose crawling strategies depending on whether there are 2 subdirectories or 75,000. Especially for the backup case, having the inode number outright would also be useful, as it helps identify hardlinks (which may skip reading the data twice) without the cost of the lstat.

See e.g. this topic in golang-nuts for some speed comparisons. This can make a very big difference.

[ianlancetaylor]

This is inherently very system dependent, so I think the question is whether it would be better to use golang.org/x/sys/unix.ReadDirents instead.

[israel-lugo]

This is inherently very system dependent, so I think the question is whether it would be better to use golang.org/x/sys/unix.ReadDirents instead.

The thing is, golang.org/x/sys/unix.ReadDirent is terribly low level in its signature. It modifies a byte buffer in place, and requires parsing each entry. The user would need to basically reimplement Readdirnames and ParseDirent; that's some 80 lines of non-trivial and error-prone code, dealing with a bunch of byte offsets etc, versus just being able to use dirInfo, err := f.ReaddirExt().

Regarding the system-dependent point, I would note that POSIX specifies d_ino, i.e. any POSIX system will at the very least tell you a file's inode, along with its name. That information in itself can be very useful, as mentioned above to efficiently identify hard links in a large tree.

Even the file type d_type is widely supported; common filesystems on Linux have long supported this, e.g. ext4, xfs, btrfs. And in case of no support, returning DT_UNKNOWN would be transparent for the application.

Perhaps even Windows may have similar information that can be returned from a directory listing. It's frankly been over a decade since I've done any Windows development so I wouldn't know.

[networkimprov]

The inode is accessible in most oses as aFileInfo.Sys().(*syscall.Stat_t).Ino. This fails on Windows, so I patched my stdlib; see
https://groups.google.com/d/topic/golang-dev/raE01Fa2Kmo/discussion

Maybe we need os.FileInfo.BetterSys() to provide the d_type if available and the fileId on Windows.

[ianlancetaylor]

@israel-lugo Go runs on non-POSIX systems. The os package is intended to be platform independent.

If the current interfaces provided by golang.org/x/sys/unix are awkward, perhaps we can think about ways to improve them.

[ianlancetaylor]

@networkimprov The issue here is whether the inode is available from reading only the directory, not opening the file and not calling Stat. Is that possible on Windows?

[networkimprov]

A quick Winapi search didn't turn up a way to get fileId from a directory entry without a CreateFile() (aka open).

Perhaps we need (f *File) Readdirentries(n int) ([]DirEntry, error). DirEntry methods are Name(), Type(), Id().

On most unixen, the DirEntry object would be populated. On Windows, its methods would CreateFile, etc on first use, to avoid that for every item. That solves the performance problem for Linux, and the missing FileInfo.Sys().Ino on Windows.

[alexbrainman]

The issue here is whether the inode is available from reading only the directory, not opening the file and not calling Stat. Is that possible on Windows?

I don't know what inode is, but, from what I can gather, the closest thing on Windows is BY_HANDLE_FILE_INFORMATION.dwVolumeSerialNumber (see https://docs.microsoft.com/en-us/windows/win32/api/fileapi/ns-fileapi-by_handle_file_information ). I suspect you can get dwVolumeSerialNumber by reading directory directly (I did not look for that API). But I don't see how it is helpful, because I don't know what is the problem that we are solving.

Alex

[networkimprov]

Alex, it's the nFileIndexHigh/Low fields (aka fileId) in that structure.

[alexbrainman]

Alex, it's the nFileIndexHigh/Low fields (aka fileId) in that structure.

You are correct, it is nFileIndexHigh / nFileIndexLow. I was wrong. I spoke too quickly.

Alex

[israel-lugo]

Thank you for your help, @networkimprov and @alexbrainman. I noticed Alex asked what is an inode and what we're trying to achieve. Let me try to bridge the gap a little, with a small introduction to Unix filesystems.

Inodes

An inode, in Unix, is the metadata structure that contains the low level details of a file (or directory, which is just a particular type of file). The inode holds the file's owner, permissions, timestamp and the list of disk sectors that contain the file's data. The inode is, for all intents and purposes, the file itself.

Inodes are identified within a filesystem by a unique number. This is the "inode number", often referred to as just "the inode", which is a slight misnomer. Some filesystems (such as the very common ext4) actually have a table of inodes, i.e. a linear array of inode structures. The inode number would be an index into that table.

Directory entries

Crucially, an inode (i.e. the file) does not have a name. It is identified solely by its inode number. The mapping of filenames to inodes is kept in directories.

A directory is a file of a special type. Its data is a list of directory entries, which map individual filenames (the files and directories inside that directory) to their respective inodes. So, if directory Foo contains file Bar.txt, which has inode number 823, then Foo's data will contain a directory entry like this:

filename	inode
Bar.txt	823

Since the directory is itself a file, it also has its own inode, and its own inode number. So Foo's parent will contain a directory entry, with filename="Foo" and inode=12345. And so on, until we reach the root of the filesystem, which exists at a well known inode number (inode 2 for most Linux filesystems I know).

Hard links

Another name for this mapping of filename to inode is a "hard link". I.e. a link from the filename to the inode number.

One important thing to note is that the mapping of filename to inode can be many to one: multiple entries in multiple directories can point to the same inode (the same file). So, as we have Foo/Bar.txt pointing to inode 823, some other Splot directory can also have a directory entry like this:

filename	inode
another-name	823

So the same file (inode 823) exists in two places of the filesystem tree: "Foo/Bar.txt" and "Splot/another-name". Both names point to the same inode, so it's the same exact file. Same owner, same timestamps, same physical disk blocks of data. Opening the file through one name or the other yields the same file object, with the same data.

In this scenario, we would say the file has two hard links. Or, colloquially, that e.g. "Splot/another-name" is a hard link to "Foo/Bar.txt".

Intended goal

Go's current directory API allows us to get either all the filenames in a directory (os.(*File).Readdirnames), or full os.FileInfo objects for every file (os.(*File).Readdir). The os.FileInfo objects contain a wealth of metadata on the file, such as the file's type, permissions, timestamps.

The problem is, os.FileInfo has "too much" information. In order to get all that, it's necessary to read the actual inode for each file. So os.(*File).Readdir has to, for each file inside the directory, read that file's actual inode. That's O(N) I/O operations, with N being the number of files in the directory.

Another problem, as @networkimprov pointed out, is that os.FileInfo doesn't contain the inode number at all, or rather, a unique identifier for the file, which in Windows would apparently correspond to the nFileIndexHigh / nFileIndexLow.

What I would like to see is an intermediate API that provides the filenames, their inode/fileId numbers and (when possible) their file types.

In Unix, just listing the directory will always give you at least the the filename and the inode number. So we can get all the filenames and their inode numbers just by reading the one directory. That's already very useful for a file crawler, since it allows you e.g. to detect hard links (two files with the same inode number). If you're writing e.g. a backup software, you probably won't need to backup the same file twice.

Many modern filesystems also store more information in the directory entry: the file's type (directory, regular file, socket, symlink, etc). So in these systems, when reading a directory's entries, we can get, for each child, its filename, inode number and file type. That is even better, because it allows the file crawler to make better decisions about which files to access. Maybe you only care about the directories, or you want to treat them first (to build the directory tree). Having the file type information for all children, just by reading the parent directory, is a great time saver.

I like @networkimprov's proposal. os.(*File).Readdirentries would return an object which is populated efficiently in most Unix systems, and can be populated with a slower fallback in other systems that don't support retrieving this information in one pass.

The question here is: can we make this fast in Windows too? In Windows, what kind of information can one retrieve about a directory's children, when listing that directory? I.e. without having to open or stat each file individually. If it's possible to get this working efficiently in Windows, that would be great. If not, however, I still think it would be worth having the API, even if it does have to fall back to opening each file individually. It won't be much of a speed improvement in Windows over os.(*File).Readdir, but it can certainly make a very large difference for POSIX (Unix-like) systems.

[networkimprov]

Below is the struct you get while walking a directory in Windows. It should be fast. So we could also do for Linux what I proposed for Windows, provide DirEntry methods that lstat() on first use to retrieve metadata not found in Linux dirents.

https://docs.microsoft.com/en-us/windows/win32/api/minwinbase/ns-minwinbase-win32_find_dataa

A small fact I neglected to mention is that the Go team tries to avoid adding new APIs :-/ However this concept could deprecate os.Readdir()...

The issue title should probably be "proposal: os: Readdirentries to read directory..."
EDIT: And you'd need to spell out the specifics; feel free to cut/paste what I wrote, and amend it as nec.

[alexbrainman]

@israel-lugo

Let me try to bridge the gap a little, with a small introduction to Unix filesystems.

Thank you for spending time to explain it all. I don't know of anything similar to inode number on Windows. Probably, something like that https://devblogs.microsoft.com/oldnewthing/20110228-00/?p=11363

Go's current directory API allows us to get either all the filenames in a directory (os.(*File).Readdirnames), or full os.FileInfo objects for every file (os.(*File).Readdir). The os.FileInfo objects contain a wealth of metadata on the file, such as the file's type, permissions, timestamps.

The problem is, os.FileInfo has "too much" information. In order to get all that, it's necessary to read the actual inode for each file.

Windows version of os.(*File).Readdirnames just calls os.(*File).Readdir, gets all filenames, and throws away the rest of os.FileInfo data. See

go/src/os/dir_windows.go

Lines 62 to 69 in 8696ae8

	func (file *File) readdirnames(n int) (names []string, err error) {
	fis, err := file.Readdir(n)
	names = make([]string, len(fis))
	for i, fi := range fis {
	names[i] = fi.Name()
	}
	return names, err
	}

os.(*File).Readdir is implemented by calling FindFirstFile and FindNextFile Windows API (like @networkimprov already explained). These functions return WIN32_FIND_DATAW structure https://docs.microsoft.com/en-us/windows/win32/api/minwinbase/ns-minwinbase-win32_find_dataw that provide everything we need. Perhaps there are other Windows APIs that provide less file information, but I am not sure, they will be faster to use.

So I don't see how any new os API can improve speed on Windows.

Alex