Expose split_tags to users

[koheiw]

When texts are full of random characters as a result of OCR, tokenization slows down because of the special character preservation mechanism in preserve_special().

We can mitigate the problem by split_hyphens = TRUE but there is no official way to disable preservation of social media tags. It causes a lot of false matches when texts are not clean.

It is only to expose the hidden split_tags:

quanteda/R/tokenizers.R

Lines 69 to 72 in 353bfe3

	if (!split_tags) {
	if (verbose) catm(" ...preserving social media tags (#, @)\n")
	regex <- c(regex, username, hashtag)
	}