bufio: Support empty final token from a SplitFunc

[spakin]

This is in reference to the golang-nuts thread How to return an empty final token from a bufio.SplitFunc.

Problem: It's not currently possible to write a "pure" bufio.SplitFunc function that either (a) returns an empty final token or (b) wants to discard all text beyond some point in the input stream. Instead, a package must implement a bufio.SplitFunc as a method on a state object, as in @robpike's demonstration of how to treat newlines as separators.

Proposed solution: Modify the implementation of Scanner.Scan to check the concrete type of the error returned by the bufio.SplitFunc function. If it's some sentinel value such as io.EOF or, say, a new bufio.EndOfScan, Scanner.Scan should store the returned token, as it does in the normal, non-error case, and return false. A subsequent scan could then continue reading the input stream from where the previous scan left off, making the proposed solution additionally useful for parsing a block of one token type followed by a block of a different token type. The proposed change to Scanner.Scan should not affect any existing code, especially if a new Error type like bufio.EndOfScan is used as the sentinel.

— Scott

[robpike]

Needs to be thought about carefully but I observe that the test case in scan_test.go for this will loop forever if the explicit counter is removed from the loop.

[gopherbot]

CL https://golang.org/cl/14924 mentions this issue.