proposal: encoding/base64: add AppendEncode and AppendDecode methods to *Encoding

[valyala]

The problem

Currently encoding/base64.Encoding provides not very convenient Encode and Decode methods when working with dst byte slices. These methods require the caller to pre-allocate dst byte slice with the space required for encoding or decoding to succeed. This significantly complicates the resulting code, which should just append base64-encoded (or -decoded) value to the provided dst byte slice. For instance:

// AppendB64Encoded appends base64-encoded src to dst and returns the dst.
func AppendB64(e *base64.Encoding, dst, src []byte) []byte {
    // the caller should make this memory allocation or further complicate the code
    // with byte slice hacks in order to avoid the memory allocation.
    buf := make([]byte, e.EncodedLen(len(src)))
    e.Encode(buf, src)
    return append(dst, buf...)

The solution

It would be great if encoding/base64 would provide AppendEncode and AppendDecode methods on the Encoding with the following signatures:

// AppendEncode appends encoded src to dst and returns the dst (which may be newly allocated).
//
// The encoding pads the output to a multiple of 4 bytes, so AppendEncode is not appropriate for use
// on individual blocks of a large data stream. Use NewEncoder() instead
func (e *Encoding) AppendEncode(dst, src []byte) []byte

// AppendDecode appends decoded src to dst and returns the dst (which may be newly allocated).
//
// If src contains invalid base64 data, it will return CorruptInputError.
// New line characters (\r and \n) are ignored
func (e *Encoding) AppendDecode(dst, src []byte) ([]byte, error)

This would simplify and optimize third-party code that currently uses Encode and Decode methods of Encoding struct.

Standard go packages already contains similar Append* functions - see strconv.Append* for details.

[valyala]

Prepared the CL

[gopherbot]

CL https://golang.org/cl/37639 mentions this issue.

[slrz]

The appending functions in strconv allow callers to avoid the string allocation. That's not necessary with encoding/base64: Encode and Decode already operate on caller-provided buffers.

This change wouldn't allow you to do something you can't do today. It's a minor convenience issue which I don't think warrants additional API surface.

[valyala]

The appending functions in strconv allow callers to avoid the string allocation. That's not necessary with encoding/base64: Encode and Decode already operate on caller-provided buffers.

The current API requires at least three additional steps from the caller in order to use Encode or Decode:

// 1. Determine the required dst buffer size:
dstLen := enc.EncodedLen(len(src))

// 2. Allocate the dst buffer
dst := make([]byte, dstLen)

// 3. Encode src into dst
enc.Encode(dst, src)

// 4. Use dst. Usually this means appending dst to another buffer
someBuf = append(someBuf, dst...)

Compare it to the equivalent one-liner if this proposal will be accepted:

someBuf = enc.AppendEncode(someBuf, src)

Additional benefits:

• someBuf may be nil.
• it is easy to avoid memory allocations with the new API similar to strconv.Append*.

[valyala]

Below are random links to real code found on github via this search, which could benefit from AppendEncode and AppendDecode:

https://github.com/twstrike/otr3/blob/88562eb4677900a21ecfa13a8c92613dd79b505d/b64.go
https://github.com/Andals/gobox/blob/8b83ba308f0a1dae2e5640be98ccb5295c24f88c/encoding/encoding.go
https://github.com/betterjun/vs/blob/8cde8837dac2c1af83aa5ac1e050a1dae7fe2297/b64/base64.go
https://github.com/olegshs/goweb/blob/a06f03fff539c7025fc8539c3bd31d300b481d5d/session/storage/encoder/base64.go
https://github.com/morikuni/gipher/blob/5c622d38764de4fcb827fc21e0a86cd8db5f8176/ciphertext.go
https://github.com/lifezq/goutils/blob/6c6eda4caf6981242e8f0e238da91b5e7fb43a99/encode/base64.go
https://github.com/benpence/matasano/blob/68d03f220c607337047a3aac8b4cabc6659249bd/src/matasano/encoding/base64.go

IMHO, Encode+EncodedLen and Decode+DecodedLen are from C world, while AppendEncode and AppendDecode are from Go world :)

[bradfitz]

Upgraded this to a proposal. @golang/proposal-review, there is also some discussion in https://golang.org/cl/37639 but let's keep it here.

[rsc]

I think what you really want is:

// AppendB64Encoded appends base64-encoded src to dst and returns the dst.
func AppendB64(e *base64.Encoding, dst, src []byte) []byte {
    // the caller should make this memory allocation or further complicate the code
    // with byte slice hacks in order to avoid the memory allocation.
    old := len(dst)
    need := e.EncodedLen(len(src))
    for cap(dst) - len(dst) < need {
        dst = append(dst[:cap(dst)], 0)
    }
    e.Encode(dst[old:], src)
    return dst[:old+need]
}

Or something like that. It's unclear this needs to be in the standard library.

[rsc]

Based on not much interest and on the fact that one can already write this function without additional help from the standard library, I'm leaning toward declining this proposal. Objections?

[rsc]

No objections.