fix: bufio.Scanner: token too long #19662
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds a field to Batcher to configure the maximum allowed line length. If this value is non-zero, the bufio.Scanner buffer will be configured scan lines up to this length. If a line exceeds this length an ErrLineToLong error will be returned. Simplified TestBatcher_read tests and added a test case for this new scenario and to ensure lines exceeding the default bufio.MaxScanTokenSize are allowed.
rogpeppe
reviewed
Sep 29, 2020
write/batcher.go
Outdated
| } | ||
|
|
||
| // limit the initial size of the buffer to a maximum of bufio.MaxScanTokenSize bytes | ||
| scanner.Buffer(make([]byte, min(bufio.MaxScanTokenSize, maxLineLength)), maxLineLength) |
There was a problem hiding this comment.
Or just scanner.Buffer(nil, maxLineLength) and let the buffer be grown as needed when lines are read. That way when all lines are short (presumably a common case) we'll use a lot less memory.
There was a problem hiding this comment.
Good feedback – consider it done!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #19586
This PR fixes two issues.
LineReadertype.write.Batcherto support line lengths greater than the default 64KiB for a newbufio.Scannermax-line-length, to configure the maximum line length. The default is 16MB, which should cover the majority of use cases.