ngram tokenizer split on whitespace

[DeanEby]

A very simple change to have our ngram tokenizer split on whitespace instead of punctuation

[KristijanArmeni]

Thanks @DeanEby! This looks straightforward. Just a small clarification (since it was not explained in the PR/issue):

Splitting on whitespace will keep punctuation symbols to tag along with tokens/words if there's no whitespace. See:

>>> import re
>>> re.split(" +", "example, with punctuation.")
['example,', 'with', 'punctuation.']

Is that OK/expected in the context of analysis, i.e. are we fine with example and example, being treated as distinct tokens? Or is that taken care of elsewhere?

Cf:

>>> re.split(r"\W+", "example, with punctuation.")
['example', 'with', 'punctuation', '']

[DeanEby]

@KristijanArmeni I think that falls under expected for now since we are looking for explicitly copy-pasted text. It is definitely something to think about and potentially implement as a future feature, but I think splitting on whitespace is better than the existing tokenization we are doing.

[KristijanArmeni]

Ok, that makes sense. Thanks @DeanEby. Fine to merge on my end.