bufio: bufio.Write and bufio.WriteString behave differently

[cccristi07]

Go version

go version go1.23.0 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/ccristi/Library/Caches/go-build'
GOENV='/Users/ccristi/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/ccristi/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ccristi/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.23.0/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.23.0/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/ccristi/Library/Application Support/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/z2/2b0v421s24g_0d_fg2txybwm0000gn/T/go-build1898012219=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Consider the following scenario:

Using WriteString

We see that the calls to the underlying io.Writer are buffered in calls with a buffer length of 3

type X struct {
	i io.Writer
}

func (x *X) Write(p []byte) (n int, err error) {
	fmt.Println("Writing ", len(p), "bytes to underlying writer")
	fmt.Println()
	n, err = x.i.Write(p)
	if err != nil {
		return
	}
	return
}

func main() {
	x := X{os.Stdout}

	bfw := bufio.NewWriterSize(&x, 3)
	//nn, err := bfw.Write([]byte("abcdefghijklmno..."))
	nn, err := bfw.WriteString("abcdefghijklmno...")
	if err != nil {
		fmt.Println("buffered writer tried writing ", nn)
		panic(err)
	}
}

yields the following result:

Writing  3 bytes to underlying writer

abc
Written  3 bytes to underlying writer
Writing  3 bytes to underlying writer

def
Written  3 bytes to underlying writer
Writing  3 bytes to underlying writer

ghi
Written  3 bytes to underlying writer
Writing  3 bytes to underlying writer

jkl
Written  3 bytes to underlying writer
Writing  3 bytes to underlying writer

mno
Written  3 bytes to underlying writer

Using bufio.Write

func main() {
	x := X{os.Stdout}

	bfw := bufio.NewWriterSize(&x, 3)
	nn, err := bfw.Write([]byte("abcdefghijklmno..."))
	//nn, err := bfw.WriteString("abcdefghijklmno...")
	if err != nil {
		fmt.Println("buffered writer tried writing ", nn)
		panic(err)
	}
}

We get the following output:

Writing  18 bytes to underlying writer

abcdefghijklmno...
Written  18 bytes to underlying writer

What did you see happen?

bufio.WriteString buffers calls to the underlying io.Writer in chunks of len(b.buf).

bufio.Write will not buffer calls to the underlying io.Writer but rather sends the whole buffer

What did you expect to see?

I was expecting to see bufio.Write behave in a similar way to bufio.WriteString.

[gabyhelp]

Related Issues and Documentation

• bufio: NewWriter may return a writer that is not new #49446 (closed)
• bufio.Writer.Write() optimization error (and a trivial fix) #949 (closed)
• bufio.Writer.Write returns wrong number of bytes written #1045 (closed)
• bufio: Reader cannot buf content greater than 32KB #33264 (closed)
• io: optimize WriteString with a pool of buffer #28311
• bufio: small buffer size and Unicode character trigger flush at wrong time #22883 (closed)
• bufio: Writer.ReadFrom doesn't always buffer #23289 (closed)
• encoding/csv: do not use bufio.Writer in csv.Writer #33486
• Resetting a bufio.Writer seems to mutate for loop counter and loops indefinitely #47350 (closed)
• runtime: infinite loop causing stack overflow in bufio #38481 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[gopherbot]

Change https://go.dev/cl/608275 mentions this issue: bufio: make Writer.Write and Writer.WriteString behave the same when pass buffer to io.Writer

[cookieo9]

bufio.(*Writer).Write has an optimization where if its buffer is empty, and the data to be written is longer than the buffer, it performs a single Write of the original slice instead of copying data into the Writer's buffer and performing multiple writes. Since the purpose of a Write buffer is to minimize writes to the underlying writer, this makes sense and is the desired behaviour.

When the buffer is not empty, those bytes present within need to be written first, so the code fills the end of the buffer with as many bytes from the input as possible, writes that to the underlying writer, re-slices the input and checks again now that the buffer is empty if the remainder of the input is bigger than the buffer to trigger the large write behaviour, otherwise it copies the remaining input into the buffer.

bufio.(*Writer).WriteString actually has a similar optimization, but the underlying writer needs to have a WriteString method to take advantage of it. I suspect it doesn't convert the string and pass as a large buffer itself because that might copy the (potentially large) string if the cheap string->[]byte conversion fails to trigger (likely since it's calling the writer via an interface, so it can't tell if the slice escapes). In any case, if your test writer contained a WriteString method, you'd see that it is handed the full string if the bufio.Writer's buffer is empty, and thus they behave the same.

I have augmented your example to show this.

package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
)

type X struct {
	i io.Writer
}

func (x *X) Write(p []byte) (n int, err error) {
	fmt.Println("Writing", len(p), "bytes to underlying writer")
	fmt.Println()
	n, err = return x.i.Write(p)
        if err != nil {
		return
	}
	return
}

func (x *X) WriteString(s string) (n int, err error) {
	fmt.Println("Writing string with", len(s), "characters to underlying writer")
	fmt.Println()
	n, err = io.WriteString(x.i, s)
        if err != nil {
		return
	}
	return
}

func main() {
	x := X{os.Stdout}

	bfw := bufio.NewWriterSize(&x, 3)
	//nn, err := bfw.Write([]byte("abcdefghijklmno..."))
	nn, err := bfw.WriteString("abcdefghijklmno...")
	if err != nil {
		fmt.Println("buffered writer tried writing ", nn)
		panic(err)
	}
}

WriteString now prints everything via a single call to our writer's new WriteString method:

Writing string with 18 characters to underlying writer

abcdefghijklmno...

[ianlancetaylor]

@cookieo9 Thanks for the explanation.

@cccristi07 Regardless of that explanation, I don't see why this is a bug. What difference does it make?

[cccristi07]

@ianlancetaylor @cookieo9, that makes sense! Thank you for the explanation!