proposal: io: add `WriterTo` to `PipeReader` and `ReaderFrom` to `PipeWriter`

[Jorropo]

There is opportunities to skip synchronization events, allocations and maybe even do some zero copies (if WriteTo and ReadFrom are called concurrently).

If ReadFrom is called this would allow Read to acquire a reference to the Reader passed in and pass the buffer slice to the reader directly (skipping a possible allocation that would have been required to copy data from Reader to PipeWriter).
If WriteTo is called this would allow Write to acquire a reference to the Writer passed in and pass the buffer slice to the reader directly (skipping a possible allocation that would have required to copy data from PipeReader to Writer).
If ReadFrom and WriteTo are called this would allow to call io.Copy (or equivalent reusing a buffer) to maybe do a zero copy between the Writer and Reader.

The implementation would need to account for the fact that ReadFrom can be called multiple times and WriteTo need to waits until the pipe is closed, but nothing that cannot be done.

The extra channels required to pass the Writer and Reader around might be much in term of extra allocations, I guess a sync based implementation (instead of channels) would be better (as it would just embed all synchronization primitives into the pipe struct).

[ianlancetaylor]

This doesn't seem to match the general model for io.Pipe, which is that reads and writes are matched when possible.

[Jorropo]

This doesn't seem to match the general model for io.Pipe, which is that reads and writes are matched when possible.

I don't understand what this means.

I shouldn't have assumed this was obvious, but calls to WriteTo and ReadFrom would block until completion is received from the other side, exactly how Read and Write do it except it is sharing a Writer and Reader instead of []byte.

[ianlancetaylor]

What I mean is that io.Pipe is designed so that every call to Write matches a call to Read, when possible. If you don't want that behavior, there isn't much reason to use io.Pipe. So I think you're right that we could make io.Copy of an io.Pipe more efficient, but I don't understand how that would help any real program.

[Jorropo]

I think I get it now, it's that one call to write will always match one call to read (assuming the read buffer is big enough).

That seems to be a very special detail to me. When I use it, it's because I have poorly architectured code, one function wants to write the other one want to read (imagine using json.Encoder with http.NewRequest for example).

So I just throw an io.Pipe and a goroutine at it because it require order of magnitude less time than implementing a lazily json serialiser that would serialise inside .Read calls (for example, I don't often do that with json it's just an example, usually I use bytes.Buffer, I have other GiBs big formats where streaming actually matters).

However io.Pipe is not that fast for this and I was hopping to give it a bit more humpf (by skipping a copy for example).

---

Adding ReadFrom and WriteTo would not affect the 1 to 1 relationship of Write and Read.
Also ReadFrom would also have a 1 to 1 relationship with Read (altho it's more likely your buffer isn't big enough and you have to call multiple times).

[hanwen-flow]

So I think you're right that we could make io.Copy of an io.Pipe more efficient, but I don't understand how that would help any real program.

Docker liberally uses io.Readers containing tar files as part of the API (example: https://github.com/moby/moby/blob/master/daemon/graphdriver/overlay2/overlay.go#L715). These io.Readers are created by connecting a tar.Writer to io.Pipe(). If the sink to such a Reader is a file descriptor again, copy_file_range(2) could speed it up. This also requires #70807.

IMO, the Docker APIs are poorly designed, but changing them to take io.Writer as sink argument is hard.

[hanwen-flow]

Hi @ianlancetaylor, looking for your input.

Specifically, using copy_file_range on BTRFS/XFS can yield speedups of up to 1000x with the right massaging. When you say:

If you don't want that behavior, there isn't much reason to use io.Pipe

do you mean that programs should come up with their own fork of io.Pipe/io.Copy to reap this performance benefit?

[hanwen-flow]

The proposal at https://go-review.googlesource.com/c/go/+/664395 doesn't work, but can be made to work with some twiddling. However, on trying to use this, I discovered why this can probably not be added to the standard library: when copying from a file into a Pipe with io.Copy, io.Copy must call PipeWriter.ReadFrom to pass on the source file to PipeReader. However, the spec says that io.Copy prefers to call the source's WriteTo, which means that os.File tries to write into io.Pipe, which is not possible, and then falls back to the slow default.

When combining zero-copy ReadFrom/WriteTo with complicated scenarios (eg. tar zero-copy as in #70807 passed through io.Pipe and written to zero-copy tar.Writer again), it is easy for zero-copy to get broken. I think a fundamental redesign might be appropriate, as discussed in #67074 (comment)