Add TokTok tokenizer

[oxinabox]

An earlier incomplete hack at #5
exists but was never tested.

We should port it, and use the new TokenBuffer API.
https://github.com/JuliaText/WordTokenizers.jl/blob/master/src/words/fast.jl

Summary (From #5)

Source https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl
It is Apache2

See also NLTK's implementation https://www.nltk.org/_modules/nltk/tokenize/toktok.html

When this is done I think that it should be the default tokenizer.
Because multilingual and doesn't screw up URLs

This code is untested, and not yet linked in.
I just ported the perl to sed.
Well to the PCRE extended sed that we actually use.
Which doesn't involve much.

Will want to port over NLTK's tests, which are hopefully comprehensive enough to check that I didn't mess anything up

[aquatiko]

@oxinabox could you link the reference to the nltk's tests that you mentioned

[oxinabox]

I thought they would be in https://github.com/nltk/nltk/blob/develop/nltk/test/unit/test_tokenize.py
but they are not.
May have to write our own.
Probably by using NLTK to generate reference tokenizations

[aquatiko]

Also, I looked in fast.jl, It would be great if there could be some regex based lookahead function. If seems right, maybe I can work on that before

[oxinabox]

work on it as part of the same PR?

[aquatiko]

No, in a separate PR

[oxinabox]

In general most PRs either:

• add a feature that is exposed to the user
• cleanup code with no change

Adding a feature to the purely internal TokenBuffer would be unusual, particularly when there are not multiple PRs waiting on such a feature.
But I am not against it. A small PR is an easy to review PR.
It will want good unit tests

[aquatiko]

Oh I see!! No worries I will add all of it in a single PR :)