proposal: os: add iterator variant of File.ReadDir

[bradfitz]

Proposal Details

Today we have:

https://pkg.go.dev/os#File.ReadDir etc

func (f *File) ReadDir(n int) ([]DirEntry, error)

That n feels so antiquated now that we have iterators! I propose that we add an iterator-based variant. 😄

/cc @neild

[gabyhelp]

Related Issues and Documentation

• proposal: io/fs: add `fs.Files()` iterator #68983 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[bradfitz]

(Note that this is complementary to #64341, which goes recursively)

[ianlancetaylor]

You left out the most interesting part of the proposal: what should the new method be called?

Also, should it return iter.Seq2[DirEntry, error] or should it return (iter.Seq[DirEntry], error) ?

[mateusz834]

Also, should it return iter.Seq2[DirEntry, error] or should it return (iter.Seq[DirEntry], error) ?

Or even (iter.Seq2[DirEntry, error], error)

EDIT: I was thinking about the global os.ReadDir, this probably matters only there.

[gazerro]

You should be able to iterate over entries in both directory order and sorted by filename.

Having an iterator return an error can help you remember to handle errors, but it can also make the code a bit messier if error handling isn’t just about returning the error.

That said, here’s my proposal:

// DirEntries returns a collection of the contents in the directory associated
// with file. It always returns a non-nil collection. If f is nil or not a
// directory, the iterator methods return an empty sequence, and a call to Err
// will return the previously occurred error.
func (f *File) DirEntries() *DirEntries

type DirEntries struct {
	// contains filtered or unexported fields
}

func (d *DirEntries) All() iter.Seq[DirEntry]

func (d *DirEntries) Sorted() iter.Seq[DirEntry]

func (d *DirEntries) Names() iter.Seq[string]

func (d *DirEntries) SortedNames() iter.Seq[string]

func (d *DirEntries) Err() error

and used as shown below:

entries := f.DirEntries()
for entry := range entries.All() {
        // ...
}
if err := entries.Err(); err != nil {
        // handle the error
}

[mateusz834]

I personally do not like having a separate Err method, it is easy to forget to use it.

[earthboundkid]

I personally do not like having a separate Err method, it is easy to forget to use it.

That is a very fair criticism, but I am writing my own iterating wrapper around fs.WalkFunc/filepath.WalkFunc, and I have found there's not a good alternative. My current API takes an error handler callback and has a method to return the last error, but it's not totally satisfying. It's also a fairly fiddly API with a lot of decision points, and the more I work on it, I'm not actually sure it makes sense to move it into the standard library as opposed to just letting third parties handle it.

[jimmyfrasche]

iter.Seq2[T, error] makes sense when there can be an error per iteration and you can continue the loop after any such error. Otherwise, the error is only non-nil at most once and you need to write the error handling code that is logically after the loop inside the loop.

There are many places where it's possible to forget to check an error in Go. For example, if you have a iter.Seq2[T, error] you can write for t := range seq

[mateusz834]

There are many places where it's possible to forget to check an error in Go. For example, if you have a iter.Seq2[T, error] you can write for t := range seq

I thought that the conclusion of #65236 was to introduce a vet check for this, but it does not seem to be added.

#65236 (comment)

[neild]

We have entirely too many ways to list a directory's contents:

• os.ReadDir
• os.File.ReadDir
• os.File.Readdir
• os.File.Readdirnames
• io/fs.ReadDir
• io/fs.ReadDirFile.ReadDir
• io/fs.ReadDirFS.ReadDir
• io/fs.WalkDir
• path/filepath.Walk
• path/filepath.WalkDir

Inconsistencies abound: Some functions return the full directory listing, some are iterative and return a chunk, some sort results, some don't.

And yet we don't seem to have enough functions, because there are gaps: Walk is less efficient than WalkDir because it calls lstat on each file; WalkDir is less efficient than Walk because it needs to read each directory into memory to sort its contents. Walk/WalkDir do a preorder traversal, but some operations (like RemoveAll) require a postorder traversal.

This isn't necessarily an argument against adding an iterator variant of ReadDir, but I think that we need to have a clear understanding on how new directory listing functions fit into the existing mess.

(It's incredibly tempting to try to propose One New API that subsumes all the existing ones--flat directory listing, tree walking, pre- or post-order traversal, sorted or unsorted, a traversal-resistant way to stat or open files, room for expansion with whatever we forgot.)

[mateusz834]

iter.Seq2[T, error] makes sense when there can be an error per iteration and you can continue the loop after any such error. Otherwise, the error is only non-nil at most once and you need to write the error handling code that is logically after the loop inside the loop.

That is a very fair criticism, but I am writing my own iterating wrapper around fs.WalkFunc/filepath.WalkFunc, and I have found there's not a good alternative. My current API takes an error handler callback and has a method to return the last error, but it's not totally satisfying.

An idea that comes to my mind, this kind of helper can be added to the iter/errors package:

func ErrorAbsorber[T any](iter iter.Seq2[T, error], out *error) iter.Seq[T] {
	return func(yield func(T) bool) {
		for v, err := range iter {
			if err != nil {
				*out = err
				break
			}
			if !yield(v) {
				return
			}
		}
	}
}

Usage:

var err error
for _ = range ErrorAbsorber(ErrorIter(), &err) {
        // logic
}
if err != nil {
        // error handling logic
}

This might help in such cases to move the error handling logic easily from the loop body.