[test] add test for tokenizer

[nnnyt]

Thanks for sending a pull request!
Please make sure you click the link above to view the contribution guidelines,
then fill out the blanks below.

Description

(Brief description on what this PR is about)
add test to cover the tokenizer

What does this implement/fix? Explain your changes.

add test to cover the tokenizer

Pull request type

• [DATASET] Add a new dataset
• [BUGFIX] Bugfix
• [FEATURE] New feature (non-breaking change which adds functionality)
• [BREAKING] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [STYLE] Code style update (formatting, renaming)
• [REFACTOR] Refactoring (no functional changes, no api changes)
• [BUILD] Build related changes
• [DOC] Documentation content changes
• [OTHER] Other (please describe): add test

Changes

• add test to cover the tokenizer

Does this close any currently open issues?

close #81

Any relevant logs, error output, etc?

N/A

Checklist

Before you submit a pull request, please make sure you have to following:

Essentials

• [-] PR's title starts with a category (e.g. [BUGFIX], [FEATURE], [BREAKING], [DOC], etc)
• [-] Changes are complete (i.e. I finished coding on this PR)
• [-] All changes have test coverage and al tests passing
• [-] Code is well-documented (extended the README / documentation, if necessary)
• [-] If this PR is your first one, add your name and github account to AUTHORS.md

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[codecov-commenter]

Codecov Report

Merging #82 (469af6f) into dev (cc357e6) will increase coverage by 0.14%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##              dev       #82      +/-   ##
===========================================
+ Coverage   99.85%   100.00%   +0.14%     
===========================================
  Files          46        46              
  Lines        1364      1364              
===========================================
+ Hits         1362      1364       +2     
+ Misses          2         0       -2     

Impacted Files	Coverage Δ
EduNLP/Pretrain/gensim_vec.py	100.00% <ø> (ø)
EduNLP/Tokenizer/tokenizer.py	100.00% <ø> (ø)
EduNLP/utils/data.py	100.00% <ø> (ø)
EduNLP/SIF/tokenization/tokenization.py	100.00% <0.00%> (+0.87%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc357e6...469af6f. Read the comment docs.

[tswsxk]

Coverage is still not 100%, see the checks.

[tswsxk]

And push to dev rather than master

[nnnyt]

The coverage is 100% now