Replace buftok due to incorrect delimiter handling

[eagletmt]

The issue is the same as described in #481.

Instead of tweaking data to cope with buftok's bug, I implemented a similar Twitter::Streaming::BufferedTokenizer and add specs for it.
You can see the buftok's bug by replacing described_class with BufferedTokenizer in spec/twitter/streaming/buffered_tokenizer_spec.rb.

[coveralls]

Changes Unknown when pulling 88940b8 on eagletmt:replace-buftok into * on sferik:master*.

[coveralls]

Changes Unknown when pulling a69126e on eagletmt:replace-buftok into * on sferik:master*.

[sferik]

Thanks for the patch. Is this really not possible with buftok? If not, perhaps you could apply this patch to buftok itself? It looks like the project hasn’t been updated in 7 years. I can’t even find the code on GitHub. Perhaps it’s time for an update. I imagine many other projects that use buftok could benefit from this patch. I would be willing to take over maintenance of this project if @tarcieri or @martinemde are willing to give me gem push access.

/cc @halorgium

[eagletmt]

buftok is totally broken if delimiter.size > 1.

The reason I rewrote is:

• buftok doesn't seem to be maintained.
• Its code base is very small.

Rewriting is much easier than modifying for me.

I don't know how many projects use buftok actually, but I suspect that many projects implement their own tokenizer because it is too simple.
I had a look at em-twitter and found that it uses a very similar class https://github.com/eventmachine/eventmachine/blob/master/lib/em/buftok.rb (and it also seems to be broken if delimiter.size > 1).

[sferik]

@eagletmt If buftok is broken, I’d like to try to fix it, since I believe its functionality is generally useful. I am willing to do the maintenance.

I’ve created a buftok repo on GitHub that I’ve imported from the gem. If I get push access to the gem and you’re be willing to submit your pull request to that repo, we can fix the problem there.

I’m currently trying to find the specs for buftok, since they weren’t packaged with the gem and the RubyForge project seems to be gone.

[eagletmt]

Okay, I try to make a pull request to sferik/buftok.
It would contain specs like spec/twitter/streaming/buffered_tokenizer_spec.rb.

[martinemde]

Happy to give you push access, but I think @tarcieri may be the only one that still has access to this. I think I'm added as an author but not a gem pusher.

Wow what an old gem.