bufio: Read reads at most once

[JordanStoker]

What version of Go are you using (go version)?

go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Ubuntu 18.04 amd64
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jordan/.cache/go-build"
GOENV="/home/jordan/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/jordan/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jordan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/snap/go/6633"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/snap/go/6633/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build344922599=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Called bufio.Read multiple times until the next read required more bytes than was available i.e. len(buf) > bufio.Available(). When this happened, bufio copied as much as was available in the current buffer but did not attempt to read the remaining bytes requested.

https://play.golang.org/p/M953MGK136s

What did you expect to see?

I expected bufio.Read to recursively perform an io.Read until the requested buffer length was fully read.
Although the documentation does suggest this behavior it is inconsistent with its C-equivalent and the rest of this package - e.g. Discard, write (would write at most twice, w)

What did you see instead?

An incomplete read with no attempt to read the remaining buffer

[Theodoree]

func NewReaderSize(rd io.Reader, size int) *Reader { b, ok := rd.(*Reader) if ok && len(b.buf) >= size { return b } if size < minReadBufferSize { size = minReadBufferSize } r := new(Reader) r.reset(make([]byte, size), rd) return r }
the Reader is struct . not is interface.

[mattn]

You should make block with two triple-backquote ```.

```
func NewReaderSize(rd io.Reader, size int) *Reader {
    b, ok := rd.(*Reader)
    if ok && len(b.buf) >= size {
        return b
    }
    if size < minReadBufferSize {
        size = minReadBufferSize
    }
    r := new(Reader)
    r.reset(make([]byte, size), rd)
    return r
}
```

[JordanStoker]

@davecheney If I'm using a bufio.Reader to read a large data file I will get some truncated reads.

To work around this, I'd need to check for the bufio.Read output for truncated read and then call the function again... In C I wouldn't have to do this myself, the Read function would do it for me. It would be a straight forward addition to the package

[davecheney]

@JordanStoker i'm sorry, my initial comment was incorrect. I deleted it, I'm sorry.

[davecheney]

To work around this, I'd need to check for the bufio.Read output for truncated read and then call the function again... In C I wouldn't have to do this myself, the Read function would do it for me. It would be a straight forward addition to the package

The short version is, io.Reader doesn't guarentee to read anything more than 1 character, in fact, it can read 0 characters without error.

[Theodoree]

@mattn Thanks for reminding

[toothrot]

I am not sure how this could be changed with the compatibility promise.

I might be missing something, could you clarify how the documented suggestion of using io.ReadFull doesn't help in your scenario?

/cc @griesemer @bradfitz @ianlancetaylor

[JordanStoker]

@toothrot In my scenario, I am unpacking small data structures from a large file.
Loading the whole file into memory isn't feasible and reading just the bytes I need for each small structure would be very slow - too many System calls. The compromise is to read data in manageable chunks, which is where the bufio package is helpful.
However, if my next read is larger than the remaining bytes available in the bufio.Reader buffer then my data structure gets truncated.

Here is a workaround I've used where I would recursively call the bufio.Reader until my data has been fully read. Recursion is overkill in my scenario (my structures are much much smaller than the bufio buffer), a single bufio.Reader.Read would have sufficed.

func readBuf(r *bufio.Reader, p []byte) (nn int, err error) {
	nn, err = r.Read(p)
	if err != nil {
		return
	}
	if nn < len(p) {
		var n int
		n, err = readBuf(r, p[nn:])
		nn += n
	}
	return
}

For the opposite scenario where I want to pack data in a file, the bufio.Writer.Write (https://golang.org/src/bufio/bufio.go) would iteratively write the data if the data is larger than the buffer available. I would have expected the same behaviour from bufio.Reader.

[davecheney]

@JordanStoker it looks like you can replace your readBuf function with https://golang.org/pkg/io/#ReadFull as readBuf reads up to len(p) bytes or until an error occurs.