Add Tweet Tokenizer

[Ayushk4]

The tweet tokenizer has been added.

• Tokenizer added
• Documentation
• Tests

The following is how the tokenizer works:

1. The regular expressions are made for WORD_REGEX (core tokenizer), HANG_REGEX
   and EMOTICONS_REGEX.
2. The function replace_html_entities is used to replace the html_entities (eg: "Price: £100" becomes "Price: £100")
3. The string is processed optionally for reducing the length of strings (like "......" becomes "..." and "waaaaay" becomes "waaay" also the twitter handles are optionally removed.
4. The String is tokenized.
5. preserve_case by default is set to true. If it is set to false,
   then the tokenizer will downcase everything except for emoticons.

I have 2 questions related to this -

• I am currently matching the regex in replace_html_entities function twice for the ones that match the outer capturing group. Is there a way that I can use replace and pass all the matched subgroups into the convert_entity_function inside it? Refer https://github.com/Ayushk4/WordTokenizers.jl/blob/853331c43a1122690fdd32a78fd040034433c8d0/src/words/tweet_tokenizer.jl#L188 and 132-134 lines on the same file.

• Should we host documentation for this repository similar to https://juliatext.github.io/TextAnalysis.jl/? I feel that this will grow as more tokenizers will be added, so maybe we can have examples on each.

[codecov-io]

Codecov Report

Merging #13 into master will decrease coverage by 4.98%.
The diff coverage is 72%.

@@            Coverage Diff             @@
##           master      #13      +/-   ##
==========================================
- Coverage   80.86%   75.88%   -4.99%     
==========================================
  Files           9       10       +1     
  Lines         277      651     +374     
==========================================
+ Hits          224      494     +270     
- Misses         53      157     +104

Impacted Files	Coverage Δ
src/split_api.jl	0% <ø> (ø)	⬆️
src/words/fast.jl	81.81% <66.66%> (+0.56%)	⬆️
src/words/tweet_tokenizer.jl	72.04% <72.04%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d877dc...c7bd296. Read the comment docs.

[oxinabox]

for record keeping purposes this is to address #3

[oxinabox]

Should we host documentation for this repository similar to https://juliatext.github.io/TextAnalysis.jl/? I feel that this will grow as more tokenizers will be added, so maybe we can have examples on each.

My feeling is that tokenizers will remain simple enough that they can all go in the readme.
At the point in which this changes, then we can look at a solution like Documenter.jl and a documentation page. But I find in general dealing with that is a surprisingly constant source of complexity to a project.

[oxinabox]

Cool!
I've just done a first pass for style and basic optimisation.
Looks pretty Ok.
but I'll do another more careful pass over once these changes are made,
and once tests are written.

All globals need to be declared const.

You seem to use a lot of inner functions which are called only once,
and I am not seeing much gain in clarity from them.
The functionality could be moved to where it is used,
or they coukd be made outer functions
as suits.

[Ayushk4]

Now only the tests and documentation part remain :)

[Ayushk4]

I have made the suggested changes, you may review this PR.

[oxinabox]

Last tiny things then we can merge this.

[Ayushk4]

The changes have been made and pushed. The CI tests also pass.

[oxinabox]

🎉