bufio: Scanner considers a buffer of length = len(token) as not enough

[k0nst4nt1n]

What version of Go are you using (go version)?

$ go version
go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

Haven't tried

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/k0nst4nt1n/Library/Caches/go-build"
GOENV="/Users/k0nst4nt1n/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/k0nst4nt1n/_gopath/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/k0nst4nt1n/_gopath"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3q/jr3whwkn13ld3j1s1xms0v4w0000gp/T/go-build070878361=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Golang bufio.Scanner considers a buffer of length = len(token) as not enough, please see example below:
(full code is here https://play.golang.org/p/3e95HG3j4H-)

func main() {
	d1 := "12" // data size == 2
	r1 := strings.NewReader(d1)
	
	s1 := bufio.NewScanner(r1)
	s1.Split(bufio.ScanLines)
	s1.Buffer(make([]byte, len(d1)), len(d1)) // buf size == 2
	
	_ = s1.Scan()
	fmt.Printf("Token=%s, Err=%v\n", s1.Text(), s1.Err())
}

What did you expect to see?

No error (output: Token=12, Err=<nil>)

What did you see instead?

Error (output: Token=, Err=bufio.Scanner: token too long)

[ianlancetaylor]

I don't see an easy fix here. The code would have to do a look ahead when it reaches the end of the buffer, which would be complicated, and would help approximately nobody.

This should probably just be mentioned in the documentation.

[PawelPru]

You are using
s1.Split(bufio.ScanLines)
//scanlines add additioonl byte for endline char "\n"

change to
s1.Split(bufio.ScanBytes)

or add
s1.Buffer(make([]byte, len(d1)), len(d1)+1) // buf size == 2

[k0nst4nt1n]

bufio.ScanLine

[k0nst4nt1n]

I know how to workaround with +1 and bufio.ScanBytes is not what I want. I want to read the whole line until the EOL or EOF.

[PawelPru]

replace string to string with EOL to simulate reading file
d1 := "12\n"

[k0nst4nt1n]

replace string with EOL to simulate reading file
d1 := "12\n"

The issue is not about how to workaround or simulate something. I can make it work. The issue emphasizes how EOL and EOF are treated differently in terms of how big your buffer should be for the scanner. This behavior is not immediately obvious (and of course it doesn't matter what we read, a file can also not have EOL).

[PawelPru]

scanline without EOF and EOL is not working , take a look here
https://golang.org/src/bufio/scan.go?s=11967:12045#L340

I think you would like to add additional "if" to scanline function to read string without EOF and EOL

[k0nst4nt1n]

scanline without EOF and EOL is not working , take a look here
https://golang.org/src/bufio/scan.go?s=11967:12045#L340

I think you would like to add additional "if" to scanline function to read string without EOF and EOL

I'm not sure why you say "without EOF and EOL" there is EOF in my case. In this case EOF is just a contract supported by the Reader interface. No more data to read it will return EOF.