proposal: os: add FS() (filesystem simply using os functions)

[twmb]

With the advent of fs.FS, I prefer to use that interface where possible. One paper cut that I've run into a few times is that there no way to use the os package outright as an fs.FS. Today, there is only os.DirFS, which returns a filesystem anchored at the given directory.

This is great for something like a file server, where we only want to serve files under a given directory and not provide access to anything outside that directory, but is less great for when we want to provide arbitrary access to the file system. In particular, if I have a program that accepts flags for where to load files, I want to accept both a local path (cmd -path localfile), or a fully specified path (cmd -path /absolute/directory/file). Where I use fs.FS within this program, I have to specify a thin wrapper around os, because DirFS does not provide the necessary flexibility.

The proposal is to add the following function and implementation:

var osFS = new(hostFS)

func FS() fs.FS { return osFS } // or `return new(hostFS)` and nix the global var

type hostFS struct{}

func (*hostFS) Open(name string) (fs.File, error)          { return Open(name) }
func (*hostFS) ReadDir(name string) ([]fs.DirEntry, error) { return ReadDir(name) }
func (*hostFS) Stat(name string) (fs.FileInfo, error)      { return Stat(name) }

[mpx]

These functions already exist in a different form:

import "io/fs"
...
root := os.DirFS("/")
...
f, err := root.Open(path)
dirs, err := fs.ReadDir(root, path)
fi, err := fs.Stat(root, path)

Note, all fs.FS paths are "unrooted" so there is still a fundamental difference with os.Open, etc.. The docs for fs.ValidPath have more details.

[twmb]

The different form is exactly what is the limiting factor today, which is that DirFS is unrooted and does not allow access outside of the initial directory. If I start with os.DirFS("/"), then that is not the directory I am currently in -- os.Open handles either a rooted path or a cwd path.

Given the current unrooted aspect, this proposal is likely asking to relax that restriction. There's likely a good reason for the restriction that I'm unaware of, though.

[mpx]

From the draft design:

The use of unrooted names—x/y/z.jpg instead of /x/y/z.jpg—is meant to make clear that the name is only meaningful when interpreted relative to a particular file system root, which is not specified in the name. Put another way, the lack of a leading slash makes clear these are not host file system paths, nor identifiers in some other global name space.

OS paths and fs.FS paths are fundamentally different. fs.FS doesn't have concepts like "relative to current working directory" and "OS root directory". Relaxing the API to accept "/" would lead to the confusion described above. #42716 exists to make these differences clearer to developers.

It's probably better/less buggy to translate paths to fs.FS style before using the API. Eg, convert to an absolute path and remove the "/" prefix. Perhaps this translation functionality could be added to the os package?

[josharian]

Somewhat related: #44279.

[mpx]

#44286 also highlights some difficulty using relative paths.

[twmb]

Perhaps this translation functionality could be added to the os package?

This runs into the problem of dealing with the current working directory, which would be at a different path prefix. Rereading the proposal and ValidPath documentation, maybe fs.FS is not meant to replace os in many use cases (e.g. CLIs), which is an unfortunate limitation.

[earthboundkid]

I wanted to write an FS for CLIs that would be able to open files or URLs, e.g. my-cli -open example.txt or my-cli -open http://example.com . I ran into the problem that there's no great way to do the translations, especially around absolute paths. In the end, I think the trouble was an FS represents a collection of files, but a CLI probably only wants one file (e.g. an io.ReadCloser), so it's not a good fit.

[rsc]

fs.FS very clearly disallows paths starting with / or with ../ or even with ./ - they must be rejected by any valid Open implementation.
That ship has sailed and cannot be turned back.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[twmb]

Is there any way to make fs.FS broadly useful for rooted filepaths, or is that a nonstarter? If it's a nonstarter, this proposal should be closed.

[mpx]

Existing fs.FS implementations are incompatible with "rooted" paths, I can't see how this could be changed without breakage.

This issue, and a number of issues linked here (#44279 in particular) highlight the difficulty of translating between OS and fs.FS paths.

It may make sense to provide functionality to sanitise OS paths so they are suitable for fs.FS. Eg, via some combination of filepath.Abs and filepath.Rel. This is a very common need for anyone wanting to use fs.FS to abstract working with a collection of files.

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group