bytes: bytes.Reader returns EOF on zero-byte Read, which doesn't conform with io.Reader interface documentation

[metala]

What version of Go are you using (go version)?

$ go version

go version go1.14.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"

GOHOSTARCH="amd64"

GOHOSTOS="linux"

What did you do?

I was deserialising binary data and I hit an unexpected io.EOF when reading zero bytes. Here is a minimal example that illustrates the problem.

package main

import (
	"fmt"
	"bytes"
	"encoding/binary"
)

func main() {
    r := bytes.NewReader([]byte{0,0,0,0})
    
    var length uint32                                                               
    err := binary.Read(r, binary.BigEndian, &length)                   
    if err != nil{
        panic(err)
    }   
    fmt.Printf("length = %d\n", length)
    
    rest := make([]byte, length)
    _, err = r.Read(rest)
    fmt.Printf("error = %s\n", err)
}

What did you expect to see?

length = 0
error = %!s(<nil>)

What did you see instead?

length = 0
error = EOF

On conformity with io.Reader and other standards

An excerpt from the documentation of io.Reader with the important parts emboldened:

type Reader interface {
   Read(p []byte) (n int, err error)
}

Implementations of Read are discouraged from returning a zero byte count
with a nil error, except when len(p) == 0. Callers should treat a return of
0 and nil as indicating that nothing happened; in particular it does not
indicate EOF.

An excerpt from the man page of read(3):

This volume of IEEE Std 1003.1-2001 requires that no action be taken for read() or write() when nbyte is zero. This is not intended to take precedence over detection of errors (such as invalid buffer pointers or file descriptors). This is consistent with the rest of this volume of IEEE Std 1003.1-2001, but the phrasing here could be misread to require detection of the zero case before any other errors. A value of zero is to be considered a correct value, for which the semantics are a no-op.

Related issues

It looks like this issue is in stark contrast to Go2 proposal issue #27531.
There is also an interesting discussion (#5310) about returning (0, nil) from a Read(p []byte).

[metala]

@gopherbot please remove Documentation

[ianlancetaylor]

It seems to me that the behavior of bytes.Reader is permitted by the documentation of io.Reader. The data is at EOF, after all. I don't see anything in the documentation of io.Reader that prohibits returning 0, io.EOF in such a case.

You are asking for a special case in bytes.Reader.Read: if the caller asks for zero bytes, always return 0, nil. And if we want to make this the required behavior for io.Reader, you are asking for a special check for a read of zero bytes in many other readers as well. I think we would need a pretty convincing argument to require existing readers to change in that way.

[metala]

I am hoping for a discussion on whether this behaviour is expected or natural. Getting an EOF error on zero-byte read looks like a replacement for a missing EOF() method.

However, I agree that changing multiple Read methods is not a favourable option, because it could break a lot of stuff.
If you think that there couldn't be any discussion on the topic, we can close this issue and leave it for future reference. It is a case when the result (0, nil) makes sense.

Edit.
The reason I even posted the issue, is because it felt wrong that I need to change only the last if-statement, by removing the err != nil || check and leaving only the len(field) != fieldLen, at the last field in the deserialisation function.

[ianlancetaylor]

I think there can be discussion on the topic. My comment above was my attempt at discussion, by pointing out the issues that I found relevant. Sorry if I seem to be preventing discussion.

[metala]

Is there a reason why there is no EOF() bool method in Reader structs in the standard library?
I think it is why zero-length Read would return an io.EOF, because it's the only way to check if you are at the end, except for pos, _ := Seek(0, current); pos == Len() or Size(), which is probably not available for all Reader structs.

Add.
I will reluctantly agree that the documentation does not prohibit (0, io.EOF) in this case, but it was my take on "except when len(p) == 0", which made me think that it's just natural to have a (0, nil) on a zero-byte Read().

[ianlancetaylor]

On something like a Unix pipe, an EOF method can only be implemented by calling read, in which case we need to have somewhere to store the data and return it on a subsequent Read. Simpler to not require the Reader to handle that, and push the issue onto the caller.

[metala]

As far as I remember, read(fd, buf, len) = 0, when there is an EOF. The errno is not set and there is not even a EEOF errno.
But I see your point that performance-wise it's better to handle syscall-dependant streams this way.

[metala]

Read()-ing seems to be inconsistent in the standard library:

$ cat main.go 
package main

import (
	"os"
	"fmt"
)

func main() {
	buf := make([]byte, 0)
	n, err := os.Stdin.Read(buf)
	fmt.Printf("n=%d, err=%s\n", n, err)
}
$ go run main.go  < /dev/null
n=0, err=%!s(<nil>)

In the bytes.Reader the last position is interpreted as io.EOF, but os.Stdin.Read that makes the syscall read(0, buf, 0) = 0, is interpreted as nil error.

[davecheney]

The io.Reader interface is like no other in the Go ecosystem. Read is the only method where the caller must examine the other values returned from a function/method call before examining the error value.

[metala]

It is sad that I had to figure it out in runtime. When writing the unit tests, I was not expecting the last field to have zero length, but that's on me.

[as]

It seems incorrect to make the EOF condition directly a function of the input's length. Your example reads from a data source that contains no data. It reads 0 bytes from that data source and triggers the end of file condition because it knows that the next read will also return io.EOF.

To me it does not make sense for the first read to be successful on a data source that contains nothing. The same Read would not return io.UnexpectedEOF if your slice was 100 bytes. It would still return io.EOF. The length of the actual input slice provided to the reader has no effect on the returned error.

[metala]

It seems incorrect to make the EOF condition directly a function of the input's length. Your example reads from a data source that contains no data. It reads 0 bytes from that data source and triggers the end of file condition because it knows that the next read will also return io.EOF.

To me it does not make sense for the first read to be successful on a data source that contains nothing. The same Read would not return io.UnexpectedEOF if your slice was 100 bytes. It would still return io.EOF. The length of the actual input slice provided to the reader has no effect on the returned error.

I appreciate the way the discussion goes. Let's say we want the Reader to report io.EOF as soon as it is aware of it and let the caller handle it. But then we also care about consistency.
In Go a byte slice source in a bytes.Reader would trigger io.EOF on the last position. But reading from an empty Linux file or stdin will return (0, nil).

And then there is this:

$ cat main3.go 
package main
import "fmt"
func main() {
	data := []byte{0}
	s := data[1:1]
	fmt.Printf("%#v\n", s)
}

$ go run main3.go 
[]byte{}

A zero-length slice that is out-of-bound by a byte is just an empty slice, which I find to be similar to reading zero bytes from the end of a bytes buffer. It's kind of the no-op I expect, when making zero-length reads...