bufio: small buffer size and Unicode character trigger flush at wrong time

[ooghry]

What version of Go are you using (go version)?

go version go1.9.2 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\gopath
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

What did you do?

https://play.golang.org/p/wR1TryoaVK
Hi there
If create new buffer writer with a size of 3. then write 1byte character and then write 2byte character.
golang trigger flush only the first(1byte) character and put second character in buffer.
I expect flush all 2 characters.

What did you expect to see?

Buffered 0
Buffered 1
Buffered 3
Available 0
writer: "aب"
Buffered 1
Buffered 2
writer: "cd"

What did you see instead?

Buffered 0
Buffered 1
writer: "a"
Buffered 2
Available 1
Buffered 3
writer: "بc"
Buffered 1
writer: "d"

[robpike]

I admit the behavior is a little odd but so is the program. Also, there is no violation here of any contract, as the package does not specify when the writes will occur.

I see no actual problem here. It is a complaint about internal details of a package that is working according to the documented behavior.

[ooghry]

I fix it with this change:
https://github.com/ooghry/go/commit/40c953593febd83b0f07c7ea8ae2259a03781aa4

[bradfitz]

Closing, as this is working as intended. It's a bytes.Buffer, not a runes.Buffer. It is totally unaware of the encoding of things you put inside of it, whether that's UTF-8 or anything else.

[bradfitz]

Sorry, I too quickly assumed this was about UTF-8 boundaries.

[ooghry]

@bradfitz could I push this change https://github.com/ooghry/go/commit/40c953593febd83b0f07c7ea8ae2259a03781aa4 to repository?
(by the following rules in golang.org/doc/contribute.html)

[bradfitz]

Yes you may. No promises it'd be accepted but that'd be the next step to get a review. Be sure to include a test.

[gopherbot]

Change https://golang.org/cl/79920 mentions this issue: bufio: small buffer size and Unicode character trigger flush at wrong time

[gopherbot]

Change https://golang.org/cl/80095 mentions this issue: bufio: small buffer size and Unicode character trigger flush at wrong time

[ianlancetaylor]

The code in CL 79920 is slightly less efficient than the existing code, and I don't see any reason that the existing code is wrong. The package doesn't make any promise that the buffering is exact, and indeed even with CL 79920 it remains inexact in that it doesn't break multi-byte runes across buffer boundaries.

Why should we change this?

[bradfitz]

The old code was over-aggressively flushing on runes that encoded to fewer than 4 bytes. For instance, ASCII rune writes would flush 3 bytes too early.