bufio: Scanner.Scan() behavior changed for empty reader

[nathan-osman]

Go version

go version go1.22.2 linux/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.22.2'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3317454097=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I have created a simple test program that reproduces the issue:

package main

import (
	"bufio"
	"fmt"
	"runtime"
	"strings"
)

func main() {
	fmt.Printf("Go version: %s\n", runtime.Version())

	s := bufio.NewScanner(strings.NewReader(""))
	s.Split(func(data []byte, atEOF bool) (advance int, token []byte, err error) {
		err = bufio.ErrFinalToken
		return
	})
	fmt.Println(s.Scan())
}

^{Playground link: https://go.dev/play/p/HXAIfzWJ_cN}

What did you see happen?

Using Go 1.22.2, I get the following output:

Go version: go1.22.2
false

But earlier versions of Go have the following output:

Go version: go1.21.9
true

A bit of digging seems to point to this commit: bc2124d — however, the updated documentation really isn't clear about what return value should be expected under this scenario.

"[Scan] returns false when there are no more tokens"

"...and a nil token with ErrFinalToken immediately stops the scanning."

What did you expect to see?

I expected to see identical behavior between versions.

[seankhliao]

given the context in #56381, the new behavior seems correct. is there any reason to believe otherwise?

[ianlancetaylor]

This seems to be exactly the behavior described the release note, in the bufio section of https://go.dev/doc/go1.22#minor_library_changes. Closing as working as intended.