Migrate to regex package

[dveselov]

For now tokenizer uses built-in re package for splitting sentences into separate words, but it can't handle some special cases, but regex package can do it right:

>>> text = 'тест ... 1 одiн, Леве́к'
>>> [x.group(0) for x in re.finditer('(\w+)', text)]
['тест', '1', 'одiн', 'Леве', 'к']
>>> [x.group(0) for x in regex.finditer('(\w+)', text)]
['тест', '1', 'одiн', 'Леве́к']

[dveselov]

May be it not good idea, because regex package is C-based and PyPy platform'll get significant performance reduction.

[dveselov]

Better solution found: prepare your texts (remove diacritics and so on) before passing it to parser, like I'm do in Pinkerton