Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bufio: add Writer.AvailableBuffer #47527

Closed
dsnet opened this issue Aug 4, 2021 · 5 comments
Closed

bufio: add Writer.AvailableBuffer #47527

dsnet opened this issue Aug 4, 2021 · 5 comments

Comments

@dsnet
Copy link
Member

@dsnet dsnet commented Aug 4, 2021

There are many APIs in Go that append to a []byte and return a []byte.

Unfortunately, these APIs operate poorly with bufio.Writer since that type only accepts []byte as an input (via the Write method) and never provides a []byte. This requires logic trying to combine append-like functionality with bufio.Writer to be responsible for allocating a buffer, when bufio.Writer already has a buffer.

I propose the addition of a new method that provides access to the underlying []byte:

// AvailableBuffer returns an empty buffer with b.Available capacity.
// This buffer is intended to be appended to and
// passed to an immediately succeeding Write call.
func (b *Writer) AvailableBuffer() []byte {
    return b.buf[b.n:][:0]
}

Example usage:

var bw *bufio.Writer = ...
b := bw.AvailableBuffer()
b = strconv.AppendUint(b, ...)
bw.Write(b)

In the common cases where bw.Available() is large enough, strconv.AppendUint appends to the buffer without allocating. When the buffer is passed to Write, the underlying call to copy returns immediately without actually copying since the source and destination are identical. This saves us both an allocation and a copy.

In the rare case where it allocates, the subsequent Write call still works as expected, but needs to copy.

The detriment of this API is that it exposes the internal buffer of bufio.Writer when it was previously impossible to access it. However, the other types in the package (i.e., Reader and Scanner) already provide unsafe access to the underlying buffer via the Reader.Peek, Reader.ReadSlice, and Scanner.Bytes methods.

@gopherbot gopherbot added this to the Proposal milestone Aug 4, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Aug 4, 2021
@rsc rsc moved this from Incoming to Active in Proposals Aug 4, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Aug 4, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc
Copy link
Contributor

@rsc rsc commented Aug 11, 2021

This seems reasonable. Does anyone object to this?

@rsc rsc moved this from Active to Likely Accept in Proposals Aug 18, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Aug 18, 2021

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc rsc moved this from Likely Accept to Accepted in Proposals Aug 25, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Aug 25, 2021

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: bufio: add Writer.AvailableBuffer bufio: add Writer.AvailableBuffer Aug 25, 2021
@rsc rsc removed this from the Proposal milestone Aug 25, 2021
@rsc rsc added this to the Backlog milestone Aug 25, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 27, 2021

Change https://golang.org/cl/345569 mentions this issue: bufio: add Writer.AvailableBuffer

@gopherbot gopherbot closed this in 0d8a4bf Sep 12, 2021
dsnet added a commit to tailscale/go that referenced this issue Jan 19, 2022
This adds a new Writer.AvailableBuffer method that returns
an empty buffer with a possibly non-empty capacity for use
with append-like APIs.

The typical usage pattern is something like:
	b := bw.AvailableBuffer()
	b = appendValue(b, v)
	bw.Write(b)

It allows logic combining append-like APIs with bufio.Writer to avoid
needing to allocate and manage buffers themselves and allows the
append-like APIs to directly write into the buffer for a bufio.Writer.

Fixes golang#47527

Change-Id: I9cd169f3f8e8c7cd40818caf3daf1944c826fc66
Reviewed-on: https://go-review.googlesource.com/c/go/+/345569
Trust: Joe Tsai <joetsai@digital-static.net>
Run-TryBot: Joe Tsai <joetsai@digital-static.net>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from golang.org/cl/345569)
dsnet added a commit to tailscale/go that referenced this issue Jan 19, 2022
This adds a new Writer.AvailableBuffer method that returns
an empty buffer with a possibly non-empty capacity for use
with append-like APIs.

The typical usage pattern is something like:
	b := bw.AvailableBuffer()
	b = appendValue(b, v)
	bw.Write(b)

It allows logic combining append-like APIs with bufio.Writer to avoid
needing to allocate and manage buffers themselves and allows the
append-like APIs to directly write into the buffer for a bufio.Writer.

Fixes golang#47527

Change-Id: I9cd169f3f8e8c7cd40818caf3daf1944c826fc66
Reviewed-on: https://go-review.googlesource.com/c/go/+/345569
Trust: Joe Tsai <joetsai@digital-static.net>
Run-TryBot: Joe Tsai <joetsai@digital-static.net>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from golang.org/cl/345569)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants