proposal: add an in-memory writer/seeker to io package

[orcaman]

Hi friends,

I wanted to ask for your opinion regarding adding a type to the io package for writing and seeking in-memory.

Since there is currently no stdlib solution for this, whenever someone needs to use an in-memory writer/seeker, they have to implement their own. I have recently created a repo for this - writerseeker, following a great answer I have gotten on StackOverflow.

You can check out the use case on my repo, basically, in some serverless/PaaS environments (e.g. GAE), you can't use the filesystem even if you'd like, so if a library that you are using needs a writer/seeker and expects a file for that (the only stdlib implementation), you need to implement your own in-memory writer/seeker or use a third party package like the one I mentioned above.

What version of Go are you using (go version)?

go version go1.8.3 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/orcaman/github.com/orcaman/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lc/lr_hrpk55218jm8dn3s78ww00000gn/T/go-build241123663=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

[rasky]

I wonder if bytes.Buffer could grow a Seek method instead.

[martisch]

@rasky: I think bytes.Buffer has no Seek since it is streaming in that it "throws" data away once it is read. It is not a static view on a byte slice.

[ianlancetaylor]

I have no real opinion on this, but I'll note that the bytes package would probably be a better place than the io package. The bytes package already has bytes.Reader, which is a seekable reader.

[rsc]

It's unclear that this is needed enough to be worth having in the standard library. What code takes a WriteSeeker as its input? The standard library defines io.WriteSeeker but defines nothing that expects or returns one (in contrast to io.ReadSeeker, which is expected by http.ServeContent and returned by a few other functions).

We left Seek out of bytes.Buffer to avoid being forced to keep old data around that had been written and then read back.

If we were going to add something I think we'd want to add something that implements both io.WriteSeeker and io.WriterAt, but still it's not clear that it's important enough to be here. We need to see more evidence of need.

[dsnet]

If we had well-defined interfaces for a VFS (related: #14106 #5636), then it would be interesting to have a bytes.File as an implementation of an in-memory file, but I don't think this adds anything to stdlib as things currently stand.

A version I've used for testing purposes: https://godoc.org/github.com/dsnet/golib/memfile

[orcaman]

@rsc Regarding your question about what code takes io.WriteSeeker as input - I've seen quite a few examples. This happens when someone expects a file but wants to be a bit more generic. You can view one example here: https://github.com/unidoc/unidoc/blob/3c340d1085ea5eecf0e06e8360a6e11f8828e183/pdf/creator/creator.go#L450

About the need to have something like this in the stdlib - IMHO, it could be justified if the following two conditions are satisfied:

1. There is great need for such code
2. The code is more than just 1 or 2 lines

As for (2), I think the repo I shared shows that this condition suffices.

Regarding the need, well, it's hard for me to measure such a thing, but I can guess that according to the activity/feedback on StackOverflow and on the repo I shared (8 starts in a few days since uploaded), there is at least some need.

[spf13]

related: https://github.com/spf13/afero is a VFS implementation for Go with support for an in memory file and filesystem. It has pretty broad adoption and can serve as a good prototype of a VFS implementation.

[rsc]

VFS is out of scope here - there are other long-standing issues about that.

For io.WriteSeeker in particular, there's still no clear need. It seems worth keeping outside the standard library for now.