Skip to content
Branch: master
Find file History
svlandeg and honnibal Replacing regex library with re to increase tokenization speed (#3218)
* replace unicode categories with raw list of code points

* simplifying ranges

* fixing variable length quotes

* removing redundant regular expression

* small cleanup of regexp notations

* quotes and alpha as ranges instead of alterations

* removed most regexp dependencies and features

* exponential backtracking - unit tests

* rewrote expression with pathological backtracking

* disabling double hyphen tests for now

* test additional variants of repeating punctuation

* remove regex and redundant backslashes from load_reddit script

* small typo fixes

* disable double punctuation test for russian

* clean up old comments

* format block code

* final cleanup

* naming consistency

* french strings as unicode for python 2 support

* french regular expression case insensitive
Latest commit 46dfe77 Feb 1, 2019
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
__init__.py πŸ’« Tidy up and auto-format .py files (#2983) Nov 30, 2018
examples.py πŸ’« Tidy up and auto-format .py files (#2983) Nov 30, 2018
lemmatizer.py Add Danish lemmatizer (#2184) Apr 7, 2018
lex_attrs.py
morph_rules.py πŸ’« Tidy up and auto-format .py files (#2983) Nov 30, 2018
norm_exceptions.py πŸ’« Tidy up and auto-format .py files (#2983) Nov 30, 2018
punctuation.py Replacing regex library with re to increase tokenization speed (#3218) Feb 1, 2019
stop_words.py πŸ’« Tidy up and auto-format .py files (#2983) Nov 30, 2018
tokenizer_exceptions.py
You can’t perform that action at this time.