proposal: bufio: add option to Scanner to skip tokens that are too long

[jackxbritton]

I want to scan a reader and skip any tokens that will fire bufio.ErrTooLong. Here's an example of what I have to do with the existing implementation of scanner: https://play.golang.org/p/LimudFrPRC4.

I'd like to have the option for scanner to skip tokens that are too long. I think adding a variable to scanner like skipTooLongTokens bool, set via a call to scanner.SetFlags(bufio.LSkipTooLongTokens) or something like that, would be very helpful.

[bcmills]

This proposal needs more background: the bar for adding API surface to the standard library is high.

• What is the underlying problem you're trying to solve?
• What, if any, are the available workarounds, and what is the cost of applying those workarounds?
• Why does the solution to the problem belong in the standard library, as opposed to a third-party package? (What makes it particularly universal or subtle? References to a pattern of code across multiple projects would help.)

[rsc]

Note that if you're just scanning lines (as in the playground example) and want to deal with the "too long" case yourself, you can use bufio.Reader's ReadSlice method. The whole reason Scanner was added was to punt on the "too long" handling. It would be at least ironic to bring that back.

/cc @robpike

[robpike]

As Russ said, the purpose of this API was to make it easier to write scanners, at the cost of eliminating recovery from some errors. That has worked out well and it would be a mistake to backtrack.

I prefer this proposal not be accepted.

[jackxbritton]

Sorry for the delay folks. bufio.ReadSlice does take care of the functionality I was looking for in a more elegant way than calling NewScanner whenever an error comes in, and the docs imply that it reuses the same buffer. Thanks for the feedback.