-
Notifications
You must be signed in to change notification settings - Fork 17.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: os: new Readdirentries method to read directory entries and efficiently expose file metadata #40352
Comments
This is inherently very system dependent, so I think the question is whether it would be better to use golang.org/x/sys/unix.ReadDirents instead. |
The thing is, Regarding the system-dependent point, I would note that POSIX specifies Even the file type Perhaps even Windows may have similar information that can be returned from a directory listing. It's frankly been over a decade since I've done any Windows development so I wouldn't know. |
The inode is accessible in most oses as Maybe we need os.FileInfo.BetterSys() to provide the d_type if available and the fileId on Windows. |
@israel-lugo Go runs on non-POSIX systems. The os package is intended to be platform independent. If the current interfaces provided by golang.org/x/sys/unix are awkward, perhaps we can think about ways to improve them. |
@networkimprov The issue here is whether the inode is available from reading only the directory, not opening the file and not calling |
A quick Winapi search didn't turn up a way to get fileId from a directory entry without a CreateFile() (aka open). Perhaps we need On most unixen, the DirEntry object would be populated. On Windows, its methods would CreateFile, etc on first use, to avoid that for every item. That solves the performance problem for Linux, and the missing FileInfo.Sys().Ino on Windows. |
I don't know what inode is, but, from what I can gather, the closest thing on Windows is BY_HANDLE_FILE_INFORMATION.dwVolumeSerialNumber (see https://docs.microsoft.com/en-us/windows/win32/api/fileapi/ns-fileapi-by_handle_file_information ). I suspect you can get dwVolumeSerialNumber by reading directory directly (I did not look for that API). But I don't see how it is helpful, because I don't know what is the problem that we are solving. Alex |
Alex, it's the nFileIndexHigh/Low fields (aka fileId) in that structure. |
You are correct, it is nFileIndexHigh / nFileIndexLow. I was wrong. I spoke too quickly. Alex |
Thank you for your help, @networkimprov and @alexbrainman. I noticed Alex asked what is an inode and what we're trying to achieve. Let me try to bridge the gap a little, with a small introduction to Unix filesystems. Inodes An inode, in Unix, is the metadata structure that contains the low level details of a file (or directory, which is just a particular type of file). The inode holds the file's owner, permissions, timestamp and the list of disk sectors that contain the file's data. The inode is, for all intents and purposes, the file itself. Inodes are identified within a filesystem by a unique number. This is the "inode number", often referred to as just "the inode", which is a slight misnomer. Some filesystems (such as the very common ext4) actually have a table of inodes, i.e. a linear array of inode structures. The inode number would be an index into that table. Directory entries Crucially, an inode (i.e. the file) does not have a name. It is identified solely by its inode number. The mapping of filenames to inodes is kept in directories. A directory is a file of a special type. Its data is a list of directory entries, which map individual filenames (the files and directories inside that directory) to their respective inodes. So, if directory Foo contains file Bar.txt, which has inode number 823, then Foo's data will contain a directory entry like this:
Since the directory is itself a file, it also has its own inode, and its own inode number. So Foo's parent will contain a directory entry, with Hard links Another name for this mapping of filename to inode is a "hard link". I.e. a link from the filename to the inode number. One important thing to note is that the mapping of filename to inode can be many to one: multiple entries in multiple directories can point to the same inode (the same file). So, as we have Foo/Bar.txt pointing to inode 823, some other Splot directory can also have a directory entry like this:
So the same file (inode 823) exists in two places of the filesystem tree: "Foo/Bar.txt" and "Splot/another-name". Both names point to the same inode, so it's the same exact file. Same owner, same timestamps, same physical disk blocks of data. Opening the file through one name or the other yields the same file object, with the same data. In this scenario, we would say the file has two hard links. Or, colloquially, that e.g. "Splot/another-name" is a hard link to "Foo/Bar.txt". Intended goal Go's current directory API allows us to get either all the filenames in a directory ( The problem is, Another problem, as @networkimprov pointed out, is that What I would like to see is an intermediate API that provides the filenames, their inode/fileId numbers and (when possible) their file types. In Unix, just listing the directory will always give you at least the the filename and the inode number. So we can get all the filenames and their inode numbers just by reading the one directory. That's already very useful for a file crawler, since it allows you e.g. to detect hard links (two files with the same inode number). If you're writing e.g. a backup software, you probably won't need to backup the same file twice. Many modern filesystems also store more information in the directory entry: the file's type (directory, regular file, socket, symlink, etc). So in these systems, when reading a directory's entries, we can get, for each child, its filename, inode number and file type. That is even better, because it allows the file crawler to make better decisions about which files to access. Maybe you only care about the directories, or you want to treat them first (to build the directory tree). Having the file type information for all children, just by reading the parent directory, is a great time saver. I like @networkimprov's proposal. The question here is: can we make this fast in Windows too? In Windows, what kind of information can one retrieve about a directory's children, when listing that directory? I.e. without having to open or stat each file individually. If it's possible to get this working efficiently in Windows, that would be great. If not, however, I still think it would be worth having the API, even if it does have to fall back to opening each file individually. It won't be much of a speed improvement in Windows over |
Below is the struct you get while walking a directory in Windows. It should be fast. So we could also do for Linux what I proposed for Windows, provide DirEntry methods that lstat() on first use to retrieve metadata not found in Linux dirents. https://docs.microsoft.com/en-us/windows/win32/api/minwinbase/ns-minwinbase-win32_find_dataa A small fact I neglected to mention is that the Go team tries to avoid adding new APIs :-/ However this concept could deprecate os.Readdir()... The issue title should probably be "proposal: os: Readdirentries to read directory..." |
Thank you for spending time to explain it all. I don't know of anything similar to
Windows version of Lines 62 to 69 in 8696ae8
So I don't see how any new os API can improve speed on Windows. Alex |
By the way, python have both os.scandir and os.listdir. |
From Python docs: Using |
@israel-lugo nothing will come of this unless it's turned into a proposal with a specific API. |
Sorry, didn't have bandwidth until now. I've edited the original post, please let me know what you think. |
The os package already defines Do we really need I don't see a reason to add a Given that I'm not sure |
CC @bradfitz |
The API should be cross-platform. On Windows, dirents provide different metadata than on Linux. So DirEntry's methods should encompass most of the Windows & Linux metadata. See link below. I don't think we want @ianlancetaylor the variability of dirents across platforms is why https://docs.microsoft.com/en-us/windows/win32/api/minwinbase/ns-minwinbase-win32_find_dataa cc @bcmills |
DirEntry looks like simplified version of os.FileInfo. Why do we need 2 versions of the same thing? Users need to understand this difference - DirEntry is faster to get then os.FileInfo. And that is not even true on Windows. Sorry, but I don't see the benefits. Alex |
@networkimprov This is why the Lstat method is there. With the metadata we get from Windows, we can pretty much fill out a FileInfo for free when scanning a directory. With Linux we need one Lstat to get anything outside of Name, Id and Type. Perhaps instead of Lstat the method should just be called FileInfo. I used the Lstat name to make it clear that this is not following symlinks, but the point is really to provide a FileInfo (for free on Windows, with a syscall on Linux). |
@alexbrainman DirEntry exposes a FileInfo, plus the file Id (which is not available with any API on Windows) plus the file type, which is not retrievable on Linux without making one system call for every file. Do not underestimate the "DirEntry is faster to get" point. That is 1 system call versus 500 to list a basic directory. It's the difference between taking 10s to crawl a typical user home vs taking 2 minutes with heavy I/O. |
@ianlancetaylor That's a good point, thank you. I've udpated to return a FileMode, and proposed a new
(I've renamed Inode to Id, per suggestion from @networkimprov.) This is useful e.g. when crawling a directory tree and you care about hard links (e.g. because you're backing up files, or doing some sort of batch processing). Also, we don't have any portable API to get this today (you have to assert FileInfo.Sys() and even then that doesn't work on Windows).
That is certainly a reasonable possibility, and I am not opposed to it. The user can just Stat/Lstat explicitly in case of ModeUnknown. In the current state, the proposal tries to make DirEntry a bit friendlier, like the Python os.DirEntry. I.e. what it doesn't know immediately, it will obtain. This makes sense in my mind, because we have different cases across Linux and Windows where different syscalls need to be made anyway: Windows needs a syscall to get the inode number, but it basically gives you the other fields in a FileInfo for free. Whereas Linux is the other way around. The lightweight parameter in this case lets you control the one case where the performance penalty is not predictable for the user: on Windows, |
I'd like to put this on hold until the FS proposal is worked out. We might be able to make this available with no new API. |
@rsc, could you provide a link to the FS proposal? |
I've commented on https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/g0hwjij/ @rsc, Reddit has buried the comment so you may need this link to find it. |
As author of PEP-471 and os.scandir (included in Python 3.5+), I'll say this has definitely been useful in the Python world for speeding up recursively walking directory trees, just because of how many In my tests of This is cross-platform in Python, so I don't see why it can't be in Go. If it can be made available "with no new API" via the FS proposal, all the better! Though I'm not sure what Russ has in mind there. I think that the Python DirEntry API ended up a bit too complex. I don't think the |
Thanks for the input, Ben. I think this proposal may be moot. Our task now is to help define an API for the |
A new proposal on this topic is #41188, and there's some alternatives in that thread. |
I filed #41265. It offers a new ReadDir() API for io/fs. |
Please see #41467. |
If #41467 is accepted, I think we can close this one. |
FYI #41467 doesn't address one of the requirements given above:
See also #40352 (comment) |
API
(Update: edited to turn into a proposal, add suggested API)
This is analogous e.g. to Python's os.scandir, as pointed out by @qingyunha below.
Context
Could we please have a new File-level API to list a directory's entries, which exposes the
d_type
field (syscall.Dirent.Type
) and, ideally, alsod_ino
(syscall.Dirent.Ino
)?Under Linux, certain filesystems (such as Btrfs, ext4) store the file type information in the direntry itself. This is available via the
d_type
field, which may beDT_UNKNOWN
if the file type could not be determined for some reason (e.g. no filesystem support, or weird quirks such as "." or ".."). According toman readdir(3)
, some BSDs also support this.Currently, we have
os.(*File).Readdir
, which does anlstat
on every file and does not make use of the type information, even if it's there. This makes sense given the method's signature, since it needs to find out the file's size, mode, etc.We also have
os.(*File).Readdirnames
, which reads the dirent but only returns the name portion.It would be very useful to have an intermediate method between these two, that returns not only the name, but also the file type (which may of course be
DT_UNKNOWN
), and ideally anything else it can know from the dirent, such as the file's inode number (d_ino
orsyscall.Dirent.Ino
).This would make it much easier to implement a fast/scalable file crawler (e.g. for backup software or something else). Given a directory with 100,000 entries, being able to cheaply separate subdirectories from other files while listing the directory itself lets the crawler e.g. batch up regular files for further processing, or choose crawling strategies depending on whether there are 2 subdirectories or 75,000. Especially for the backup case, having the inode number outright would also be useful, as it helps identify hardlinks (which may skip reading the data twice) without the cost of the
lstat
.See e.g. this topic in golang-nuts for some speed comparisons. This can make a very big difference.
The text was updated successfully, but these errors were encountered: