bufio: ScanLines hangs forever if it reads an enormous string with no newlines

[SwampDragons]

What version of Go are you using (go version)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/mmarsh/Library/Caches/go-build"
GOENV="/Users/mmarsh/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/mmarsh/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/8t/0yb5q0_x6mb2jldqq_vjn3lr0000gn/T/go-build577938890=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.13.3 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.13.3
uname -v: Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.14.2
BuildVersion:	18C54
lldb --version: lldb-1001.0.13.3
  Swift-5.0

What did you do?

I sent an enormous string, (> 64*1024 bytes) through the scanner. This string contained no newlines.

Here's a repro case, plus a workaround, in a golang playground:

https://play.golang.org/p/VzBG989NJtz

What did you expect to see?

In a situation where the length of the data ScanLines searches exceeds the max token size, I'd expect either an error or for ScanLines just to return what it has.

What did you see instead?

ScanLines just hangs, I think because it loops forever, returning 0, nil, nil even when it can't get any more data because the token is already maxed out.

I was able to work around this by providing the scanner with a modified ScanLines function that just returns what it has when the data provided is the max token size, even if it hasn't reached a newline. I'm not really sure what the right default functionality should be (error vs returning what it has) but I know that I was not expecting it to hang like this.

I'd be happy to write a patch if we can agree on what would be the most intuitive/appropriate default.

Thanks so much!!

[ianlancetaylor]

In this case it seems that scanner.Scan should return with a ErrTooLong error.

In fact, I think that is what happens. If I modify your playground example to https://play.golang.org/p/xBiJf4g_XLe to print scanner.Err() after the loop completes, I see the error. Your playground program keeps running because nothing ever stops the io.WriteString.

[SwampDragons]

🤦‍♀ You're completely right. Thanks. Bug exists between keyboard and chair.