proposal: io/fs: add writable interfaces

[matthewmueller]

Go 1.16 introduced the embed and io/fs packages. The current implementation is a minimal readable filesystem interface.

This is a wonderful first step towards standardizing filesystem operations and providing the community with a lot of flexibility in how we implement filesystems. I would like us to take the next step and define a writable file system interface.

(FYI: I was surprised to not find an open issue for this, but maybe I missed something. Feel free to close if that's the case!)

Problem

fs.FS can't be modified after it's defined.

func Write(fs fs.FS) {
  // We can't write to FS.
  fs 
}

Optional interfaces could be defined in user-land:

func Write(fs fs.FS) {
  // We can't rely on vfs.Writable being implemented across community packages.
  writable, ok := fs.(vfs.Writable)
}

But it suffers from the same issues that the readable filesystem interface aimed to solve: standardizing the interface across the ecosystem.

Use Cases

I'll list of few use-cases that I've come across since Go 1.16, but I'm sure the community has many more:

• A virtual filesystem that you can write to over time. Useful for file bundlers and databases that work in-memory and flush to the operating system's filesystem at the end.
• Write files to cloud storages like Google Cloud Storage or Amazon S3.

We've already seen started to see this pop up in the community around io/fs to address the problem in user-land:

• https://github.com/psanford/memfs: defines fs.MkdirAll and fs.WriteFile
• https://github.com/vedranvuk/fsex: defines file.Write, file.Seek and file.Close.

A quick search on Github will yield more community libraries: https://github.com/search?q=%22io%2Ffs%22+language%3Ago. For many of these implementations, you can imagine a useful writable implementation.

Of course, there are many other file system libraries that came before io/fs that define writable interfaces like afero and billy.

Proposal

I don't feel qualified to define an interface, I know people have thought about this much harder than I have. What I would love to see from a community member's perspective is the following:

package fs

func WriteFile(fs FS, name string, data []byte, perm FileMode) error
func MkdirAll(fs FS, path string, perm FileMode) error

Nice to Have: Be able to define if a filesystem is readable, writeable or read-writable.

func Open(fs fs.FS) (*DB, error)   // Readable
func Open(fs fs.WFS) (*DB, error)  // Writable
func Open(fs fs.RWFS) (*DB, error) // Read-Writable

Thanks for your consideration!

[coder543]

I'm currently experimenting with writing a file format encoding/decoding/mutating package that is intended to work with files that aren't guaranteed to easily fit in memory.

I would like to implement it in terms of fs.FS, which would make it so the library doesn't have to care whether these files actually exist on the local filesystem, in memory, or stored somewhere else. In point of fact, this is intended to be an archival format that distributes the contents of a single archive over a configurable number of actual files, and these files might be distributed geographically across different regions for redundancy.

This codec package doesn't want to care about supporting all the different places that the underlying files could be stored. It just wants to take in an fs.FS and a list of paths.

Additionally, to make this package more testable, using fs.FS would make it trivial to write tests without having to actually read and write files on disk.

Unfortunately, since fs.FS is read-only, I'm sitting here thinking up complicated ways that I could support fs.FS and somehow still support actually writing and mutating files on disk.

[DeedleFake]

I've got a similar issue with my p9 package. It attempts to implement a 9P client and server in a high-level way, similar to net/http, and while I'd like to rework the file abstraction to use fs.FS, it currently would result in odd things due in part to Open() returning an fs.File, which is then read-only by design. For now, what I'm leaning towards is just having a function that takes my package's filesystem type and returns an fs.FS that abstracts it away, but the read-only problem will still be there.

Maybe something like this could work?

type RWFS interface {
  FS
  WriteFS
}

type WriteFS interface {
  // Create creates a new file with the given name.
  Create(string) (WFile, error)

  // Modify modifies an existing file with the given name.
  Modify(string) (WFile, error)
}

type WFile interface {
  Write([]byte) (int, error)
  Close() error
  // Maybe also some kind of WStat() method?
}

Then the returned types would only have to expose either reading or writing methods, and the interface would just handle it transparently.

Persionally, I think that it would be a lot better if there was some way to abstract away the specific requirement of an fs.File as the returned type so that either Open(string) (*os.File, error) or Open(string) (SomeCustomFileType, error), but that would require language changes and that seems like overkill. It could be partially done with generics, such as with type FS[F File] interface { ... }, but it has some odd potential complications, and it wouldn't be fully backwards compatible at this point.

[dhemery]

To me, this looks like the minimum requirement:

package fs

type WFile interface {
    Stat() (FileInfo, error)
    Write(p []byte) (n int, err error)
    Close() error
}

type WriteFS interface {
    OpenFile(name string, flag int, perm FileMode) (WFile, error)
}

And another (ugh) for making dirs:

type MkDirFS interface {
    MkDir(name string, perm FileMode) error
}

And some helper functions for convenience:

func Create(fsys WriteFS, name string) (WFile, error) {
    // Use fsys.OpenFile ...
}

func WriteFile(fsys WriteFS, name string, data []byte, perm FileMode) error {
    // Use fsys.OpenFile, Write, and Close ...
}

func MkDirAll(fsys MkDirFS, path string, perm FileMode) error {
    // Use fsys.MkDir to do the work.
    // Also requires either Stat or Open to check for parents.
    // I'm not sure how to structure that either/or requirement.
}

[kylelemons]

I think that we should lean more heavily on io and os, rather than making top-level WFile types. In particular, I think we should basically just define some top-level functions that can fall back all the way to Open, and not have a WFile interface at all.

Summary

• ErrUnsupported for when implementations are not available
• Optional FS methods:
  • WriteFile(name, data, perm) (error)
  • OpenFile(name, perm) (File, error)
  • Create(name) (File, error)
• Optional File methods:
  • Write(data) (int, error)
  • Truncate(size) error for use when FS.Create is being emulated
    • Only used if the file has nonzero size in Create
  • Chmod(FileMode) error for use when FS.OpenFile or FS.WriteFile are being emulated
    • Only used if the mode does not match after Open
• Helpers
  • Create(fs, name): try CreateFS, then Open+Stat+Truncate
  • OpenFile(fs, name, perm): try OpenFileFS, then Open+Stat+ChmodFile
  • WriteFile(fs, name, data, perm): try WriteFileFS, then OpenFile()+writeContents
  • Write(File, data) (int, error): calls Write or returns ErrUnsupported

Detail

See sketch of an implementation here:

https://gist.github.com/kylelemons/21539a152e9af1dd79c3775ca94efb60#file-io_fs_write_sketch-go

This style of implementation appeals to me because:

• You can check for mutability of a file with a familiar type: io.WriteCloser (or just io.Writer, but File requires Close)
• File mutability, metadata mutability, and FS mutability are all orthogonal
  • Mutable file trees can store immutable files
  • Immutable file trees can store mutable files
  • Not all files in a FS need to have the same mutability constraints
  • Subs could be mutable even though the FS is not
• This keeps the "primary" top-level interfaces the same: FS and File
• Truncate is not required if Create is never used on nonzero-length files
• Chmod is not required if the permissions are correct by default (e.g. by exposing a FixedMode from your fs package)

I think the same patterns can be used to implement Mkdir and MkdirAll on a filesystem as well.

[geoffroy-perso]

I think that also having a Remove method that can work with a fs.FS interface would be a great addition (similar to the current os.Remove function).

Just a small contribution as it feels to me in the scope of this proposal, and since no one mentionned file removal 🙂

[sbourlon]

@kylelemons just to be consistent with fs.ReadDirFile, the only File extension at the moment, and iofs draft/The File interface:

type ReadDirFile interface {
  File
  ReadDir(n int) ([]DirEntry, error)
}

type WriteFile interface {
  File
  Write(p []byte) (n int, err error)
}

There is a comment from Russ Cox about this exact change as well.

[hairyhenderson]

There's an interesting package https://pkg.go.dev/github.com/hack-pad/hackpadfs that's got some interfaces defined for a writable version of io/fs... The interfaces "feel" pretty much like they should, and are effectively the same as much as what's been discussed here already. I'm in no way associated with that project, just thought it was some interesting work that might inform how writable file interfaces fit into a future release.