Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: add an in-memory writer/seeker to io package #21592

Closed
orcaman opened this issue Aug 24, 2017 · 8 comments

Comments

@orcaman
Copy link

commented Aug 24, 2017

Hi friends,

I wanted to ask for your opinion regarding adding a type to the io package for writing and seeking in-memory.

Since there is currently no stdlib solution for this, whenever someone needs to use an in-memory writer/seeker, they have to implement their own. I have recently created a repo for this - writerseeker, following a great answer I have gotten on StackOverflow.

You can check out the use case on my repo, basically, in some serverless/PaaS environments (e.g. GAE), you can't use the filesystem even if you'd like, so if a library that you are using needs a writer/seeker and expects a file for that (the only stdlib implementation), you need to implement your own in-memory writer/seeker or use a third party package like the one I mentioned above.

What version of Go are you using (go version)?

go version go1.8.3 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/orcaman/github.com/orcaman/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lc/lr_hrpk55218jm8dn3s78ww00000gn/T/go-build241123663=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

@gopherbot gopherbot added this to the Proposal milestone Aug 24, 2017

@gopherbot gopherbot added the Proposal label Aug 24, 2017

@rasky

This comment has been minimized.

Copy link
Member

commented Aug 24, 2017

I wonder if bytes.Buffer could grow a Seek method instead.

@martisch

This comment has been minimized.

Copy link
Member

commented Aug 24, 2017

@rasky: I think bytes.Buffer has no Seek since it is streaming in that it "throws" data away once it is read. It is not a static view on a byte slice.

@ianlancetaylor ianlancetaylor changed the title Proposal: add an in-memory writer/seeker to io package proposal: add an in-memory writer/seeker to io package Aug 25, 2017

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 25, 2017

I have no real opinion on this, but I'll note that the bytes package would probably be a better place than the io package. The bytes package already has bytes.Reader, which is a seekable reader.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2017

It's unclear that this is needed enough to be worth having in the standard library. What code takes a WriteSeeker as its input? The standard library defines io.WriteSeeker but defines nothing that expects or returns one (in contrast to io.ReadSeeker, which is expected by http.ServeContent and returned by a few other functions).

We left Seek out of bytes.Buffer to avoid being forced to keep old data around that had been written and then read back.

If we were going to add something I think we'd want to add something that implements both io.WriteSeeker and io.WriterAt, but still it's not clear that it's important enough to be here. We need to see more evidence of need.

@dsnet

This comment has been minimized.

Copy link
Member

commented Aug 28, 2017

If we had well-defined interfaces for a VFS (related: #14106 #5636), then it would be interesting to have a bytes.File as an implementation of an in-memory file, but I don't think this adds anything to stdlib as things currently stand.

A version I've used for testing purposes: https://godoc.org/github.com/dsnet/golib/memfile

@orcaman

This comment has been minimized.

Copy link
Author

commented Aug 29, 2017

@rsc Regarding your question about what code takes io.WriteSeeker as input - I've seen quite a few examples. This happens when someone expects a file but wants to be a bit more generic. You can view one example here: https://github.com/unidoc/unidoc/blob/3c340d1085ea5eecf0e06e8360a6e11f8828e183/pdf/creator/creator.go#L450

About the need to have something like this in the stdlib - IMHO, it could be justified if the following two conditions are satisfied:

  1. There is great need for such code
  2. The code is more than just 1 or 2 lines

As for (2), I think the repo I shared shows that this condition suffices.

Regarding the need, well, it's hard for me to measure such a thing, but I can guess that according to the activity/feedback on StackOverflow and on the repo I shared (8 starts in a few days since uploaded), there is at least some need.

@spf13

This comment has been minimized.

Copy link
Contributor

commented Sep 18, 2017

related: https://github.com/spf13/afero is a VFS implementation for Go with support for an in memory file and filesystem. It has pretty broad adoption and can serve as a good prototype of a VFS implementation.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2017

VFS is out of scope here - there are other long-standing issues about that.

For io.WriteSeeker in particular, there's still no clear need. It seems worth keeping outside the standard library for now.

@rsc rsc closed this Sep 25, 2017

@golang golang locked and limited conversation to collaborators Sep 25, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
8 participants
You can’t perform that action at this time.