The BufferedTokenizer works great with single character delimiters, but with multi-character delimiter it doesn't address the edge case of extracting the delimiter out of the internal buffer. The original discussion as well as code examples of the edge case can be found in another pull request:
Extract from buffer in edge case of multi-character delimiter
I'd recommend adding a test too.
Yeah, the test described in #338 would do nicely here. Thoughts?
@rud @johnkchow I ended up building an eventmachine replacement however you could port the tokeniser I wrote to eventmachine.
with tests: https://github.com/cotag/uv-rays/blob/master/spec/buffered_tokenizer_spec.rb
Supports delimiters, optional start of stream indicators, regular expressions and handles common 'edge cases' such as multibyte delimiters
@stakach ah, brilliant. Thanks for sharing, looks neat.
Resolved by #547