Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: os: fast, comprehensive directory API #41265

networkimprov opened this issue Sep 7, 2020 · 7 comments

proposal: os: fast, comprehensive directory API #41265

networkimprov opened this issue Sep 7, 2020 · 7 comments


Copy link

@networkimprov networkimprov commented Sep 7, 2020

This is an alternative to #41188, which seems impractical. It offers a replacement for os.Readdir() & os.Readdirnames(). We need a new API to address performance problems and missing features in the current one.

Previous discussion...

The API below meets all the requirements discussed. It provides explicit lazy-loading, ensures that all accessible fields have data from the same retrieval, and avoids an error check after every field access.

Use cases: When you need...
a) certain fields for every item, request them in ReadDirItems().
b) certain fields for some items, request them in DirItem.Load().
c) any fields of the OS defaults that aren't universal, check for them with DirItem.Has() before access.
d) implementation-specific fields, request them as above, then check for them with DirItem.Has() before access.
e) the latest data for an item, request it with DirItem.Load().

package os

// fields uint64 is a set of |'d fieldnames
// Platforms may define their own fieldnames outside a reserved range
// ReadDirItems() & DirItem.Load() ignore fieldnames not available on this OS
// OS default fields are always accessible in DirItem

type DirItem struct { ... }
func (d *DirItem) Has(fields uint64) bool   // indicates whether fields are loaded
func (d *DirItem) Load(fields uint64) error // (re-)loads fields
func (d *DirItem) Name() string             // always succeeds
func (d *DirItem) IsDir() bool              // this and other getters panic if field not loaded (a code error)
func (d *DirItem) Id() FileId
func (d *DirItem) Size() int64
func (d *DirItem) Mode() FileMode
func (d *DirItem) ModTime() time.Time
func (d *DirItem) CreTime() time.Time       // support OS-specific fields
func (d *DirItem) SameAs(*DirItem) bool     // calls .Id() for both objects; counterpart to os.SameFile

type FileId struct { ... }          // unix inode or winapi fileId; may include device info
func (id *FileId) String() string

func ReadDirItems(name string, opt uint64) ([]DirItem, error) // opt: fields | SortXyz
                                                              // if sorted, fieldnames must include sort key
func GetDirItem(name string, resolve bool) (*DirItem, error)  // does stat() or lstat()

func (f *File) ReadDirItems(n int, fields uint64) ([]DirItem, error)
func (f *File) GetDirItem() (*DirItem, error)

const (
   ItemOS    uint64 = 0 // fieldname for OS default fields
   ItemIsdir uint64 = 1 << iota

const (
   SortDown uint64 = ... // descending switch
   SortName uint64 = ...

cc @robpike @rsc

Copy link

@rsc rsc commented Sep 11, 2020

Redesigning the os.File interface is explicitly out of scope for the io/fs proposal.
We are only making a simple interface for the current world, not building a new world.

Copy link

@networkimprov networkimprov commented Sep 12, 2020

I'll retarget this proposal to the os package.

@networkimprov networkimprov changed the title proposal: io/fs: replace Readdir/FileInfo proposal: os: fast, comprehensive directory API Sep 14, 2020
Copy link

@rsc rsc commented Sep 15, 2020

Please stop. We are not redesigning the os package either.

Copy link

@networkimprov networkimprov commented Sep 16, 2020

Please stop.

Apologies if I've given offense somehow. #41188 looks like a non-starter. Could a better directory API land in the x/ repo?

(I completed the revisions I'd started above, dropping the io/fs references.)

Copy link

@rsc rsc commented Sep 16, 2020

Every person and every project has limited bandwidth for making decisions well, especially given the large amounts of time it can take to evaluate an idea fully. Redesigning the os.File API is not on our priority list right now. I'm sorry if that priority conflicts with yours, but again we have limited time and need to focus on higher-impact things.

It is of course fine to build your own APIs in a third-party package, as always.

@rsc rsc moved this from Incoming to Active in Proposals Sep 16, 2020
Copy link

@networkimprov networkimprov commented Sep 17, 2020

Then would you tag this "Proposal-Hold" instead of closing? You're gonna need it eventually :-)

Copy link

@rsc rsc commented Sep 18, 2020

A few things.

  1. Proposal-Hold is not for "gonna need eventually".
  2. It's far from clear that we're ever going to need to completely redo the entire directory API as proposed in this issue.
  3. If we did need a new API, the API here is not what we would do. If we were going to redo the API, it would be to simplify it further, not make it significantly more complex. Any new API proposal that increases complexity by double or more is a non-starter, and this looks like about 5-10X.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.