Need (internal?) tokens_split()

[kbenoit]

This would resolve the inefficiency noted here but with a general solution. I actually started writing this based on the (internal) tokens_segment() but then went for a quicker fix since I only needed an internal hyphen solution for

quanteda/R/tokens.R

Lines 221 to 225 in 0dc5b34

	if (remove_hyphens && any(stri_detect_regex(types(x), "^.+-.+$"))) {
	x <- lapply(as.list(x), function(y)
	as.character(tokens(as.character(y), remove_hyphens = TRUE))) %>%
	as.tokens()
	}

Proposal:

tokens_split(x, pattern, valuetype = c("glob", "regex", "fixed"),
  case_insensitive = TRUE, keep_pattern = FALSE)

which would be the opposite direction, otherwise similar behaviour, to tokens_compound().

[kbenoit]

BTW here I implemented a hyphen-specific version for dfm features here, probably about as efficiently it's possible to do in straight R.

But it's easier for a dfm than for tokens because a) can use column indexing from matrices and b) feature order does not matter for dfm.

[kbenoit]

To Do: Reimplement the inefficient solution to #1498 once tokens_split() is working.

[kbenoit]

Solved in #1520 via tokens_chunk().