Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Add tests for tokenizers #6
This PR adds tests for tokenizers using https://github.com/nltk/nltk/blob/develop/nltk/test/tokenize.doctest as a reference.
The test details are
Related to this, http://weitz.de/cl-ppcre/#*allow-named-registers*, cl-ppcre has support for named groups/back references. (After all, it's an Edi Weitz library!)