Rename unnest_tokens to eliminate semantic overlap/ambiguity with tidyr::unnest

[petereckley]

Currently the naming overlaps with tidyr::unnest(), which takes unwraps list columns. By contrast tidytext::unnest_tokens() acts on a character vector column. For example, tokenise() would avoid this collision of names / semantics, though at the cost of losing the sense that multiple rows are being produced for each input text field. tokenise_to_rows or tokens_to_rows addresses this, but doesn't feel very elegant. I'm afraid I don't have the solution. But there must be a name that captures both that sense, without overlapping with tidyr::unnest()?

[dgrtwo]

The name was chosen exactly to reflect the similarity to tidyr's unnest: that it unnests after tokenizing a column into a list column. (In the original version it actually used tidyr's unnest- the current solution is somewhat faster).

In any case, since it's the most commonly used function in the package, and used by us and others in many published examples, we would have needed an extremely compelling reason to change it.

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.