bufio: document SplitFunc corner cases

[yaxum62]

The current document of bufio.SplitFunc does not cover all cases and left few details buried into implementations. Such as:

1. A Scanner behave differently when a SplitFunc returns token with nil or an empty byte slice. example
2. A Scanner usually skip advanced bytes when returned token is nil. example

Here I propose few changes to doc, clearly state those behavior. Such as:

1. If a SplitFunc returns a non-nil error, returned advance and token are ignored.
2. If a SplitFunc returns a non-nil token, even if it is empty, Scanner will always yield, no matter what error a reader previously returns.
3. If a SplitFunc returns a nil token with non-zero advance, Scanner will skip those bytes.

[bcmills]

This issue seems two describe two separate issues: missing doc comments, and a requested change in behavior.

Please file those separately: documentation issues tend to be relatively easy (and uncontentious) to fix, whereas changes in behavior may require a more rigorous proposal, especially if they are not compatible with Go 1.

[yaxum62]

I have rephrase this issue to documentation, and a separate issue will be created after this is done.

[bradfitz]

@robpike, you want to clarify Scanner docs here?

[robpike]

The docs seem to me to cover the situation well enough but of course one can always explain in more detail. Detailed response:

1. If there's an error, scanning is stopping. It makes no difference what the scanner does before it returns.
2. There is no error in this case, so I don't see why it matters what the token is. The scanner advances with that token. However, I do not understand your point about previous errors. An error on the read stops the scan.
3. This one seems again very obvious to me. There is a token (being nil is irrelevant) and an advance value, so it advances.

The docs seem clear to me, unless you're arguing that a nil token is somehow special, which it is not. That could be documented but I don't believe it's necessary. Token values are irrelevant to the scanner, which just passes them on.

[yaxum62]

The different behaviors between nil and empty slice is not that obvious since those two are generally interchangeable in golang. The difference should be clearly documented here if it is supported.

And also the feature "nil token skips bytes" are not always working as expected. In some case it will terminate the scan if the reader already hits a non-nil error. example. As @bcmills suggested the doc need to be fixed before we can fix the implementation.