Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: add an in-memory writer/seeker to io package #21592

Closed
orcaman opened this issue Aug 24, 2017 · 8 comments
Closed

proposal: add an in-memory writer/seeker to io package #21592

orcaman opened this issue Aug 24, 2017 · 8 comments

Comments

@orcaman
Copy link

orcaman commented Aug 24, 2017

Hi friends,

I wanted to ask for your opinion regarding adding a type to the io package for writing and seeking in-memory.

Since there is currently no stdlib solution for this, whenever someone needs to use an in-memory writer/seeker, they have to implement their own. I have recently created a repo for this - writerseeker, following a great answer I have gotten on StackOverflow.

You can check out the use case on my repo, basically, in some serverless/PaaS environments (e.g. GAE), you can't use the filesystem even if you'd like, so if a library that you are using needs a writer/seeker and expects a file for that (the only stdlib implementation), you need to implement your own in-memory writer/seeker or use a third party package like the one I mentioned above.

What version of Go are you using (go version)?

go version go1.8.3 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/orcaman/github.com/orcaman/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lc/lr_hrpk55218jm8dn3s78ww00000gn/T/go-build241123663=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
@gopherbot gopherbot added this to the Proposal milestone Aug 24, 2017
@rasky
Copy link
Member

rasky commented Aug 24, 2017

I wonder if bytes.Buffer could grow a Seek method instead.

@martisch
Copy link
Contributor

@rasky: I think bytes.Buffer has no Seek since it is streaming in that it "throws" data away once it is read. It is not a static view on a byte slice.

@ianlancetaylor ianlancetaylor changed the title Proposal: add an in-memory writer/seeker to io package proposal: add an in-memory writer/seeker to io package Aug 25, 2017
@ianlancetaylor
Copy link
Contributor

I have no real opinion on this, but I'll note that the bytes package would probably be a better place than the io package. The bytes package already has bytes.Reader, which is a seekable reader.

@rsc
Copy link
Contributor

rsc commented Aug 28, 2017

It's unclear that this is needed enough to be worth having in the standard library. What code takes a WriteSeeker as its input? The standard library defines io.WriteSeeker but defines nothing that expects or returns one (in contrast to io.ReadSeeker, which is expected by http.ServeContent and returned by a few other functions).

We left Seek out of bytes.Buffer to avoid being forced to keep old data around that had been written and then read back.

If we were going to add something I think we'd want to add something that implements both io.WriteSeeker and io.WriterAt, but still it's not clear that it's important enough to be here. We need to see more evidence of need.

@dsnet
Copy link
Member

dsnet commented Aug 28, 2017

If we had well-defined interfaces for a VFS (related: #14106 #5636), then it would be interesting to have a bytes.File as an implementation of an in-memory file, but I don't think this adds anything to stdlib as things currently stand.

A version I've used for testing purposes: https://godoc.org/github.com/dsnet/golib/memfile

@orcaman
Copy link
Author

orcaman commented Aug 29, 2017

@rsc Regarding your question about what code takes io.WriteSeeker as input - I've seen quite a few examples. This happens when someone expects a file but wants to be a bit more generic. You can view one example here: https://github.com/unidoc/unidoc/blob/3c340d1085ea5eecf0e06e8360a6e11f8828e183/pdf/creator/creator.go#L450

About the need to have something like this in the stdlib - IMHO, it could be justified if the following two conditions are satisfied:

  1. There is great need for such code
  2. The code is more than just 1 or 2 lines

As for (2), I think the repo I shared shows that this condition suffices.

Regarding the need, well, it's hard for me to measure such a thing, but I can guess that according to the activity/feedback on StackOverflow and on the repo I shared (8 starts in a few days since uploaded), there is at least some need.

@spf13
Copy link
Contributor

spf13 commented Sep 18, 2017

related: https://github.com/spf13/afero is a VFS implementation for Go with support for an in memory file and filesystem. It has pretty broad adoption and can serve as a good prototype of a VFS implementation.

@rsc
Copy link
Contributor

rsc commented Sep 25, 2017

VFS is out of scope here - there are other long-standing issues about that.

For io.WriteSeeker in particular, there's still no clear need. It seems worth keeping outside the standard library for now.

@rsc rsc closed this as completed Sep 25, 2017
@golang golang locked and limited conversation to collaborators Sep 25, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants