extend tokenizer regex

[emphasize]

This expands the capabilities of both word and sentence tokenizer

Tests for words tokenization

Test for sentence tokenization:

import regex as re

regex = r'(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s'
text = "Dr. Müller bought cheapsite.com for 1.5 million dollars on 8.Februar, i.e. he paid a lot for it. Did he mind? Adam Jones Jr. thinks he didn't. In any case, this isn't true... Well, with a probability of .9 it isn't."
sentences = re.split(regex, text)
for sentence in sentences:
  print(sentence)
assert sentences == [
  'Dr. Müller bought cheapsite.com for 1.5 million dollars on 8.Februar, i.e. he paid a lot for it.',
  'Did he mind?',
  "Adam Jones Jr. thinks he didn't.",
  "In any case, this isn't true...",
  "Well, with a probability of .9 it isn't."]