Split on sentence and other boundaries

[lmullen]

This pull request fixes a problem with splitting on boundaries other than words. Currently, splitting on sentence boundaries returns a list with an empty character vector:

str_split("This is a sentence. So is this.", boundary("sentence"))
#> [[1]]
#> character(0)

The problem is that boundary() sets skip_word_none = TRUE by default. But if stringi:stri_split_boundaries() is called for any boundary other than word boundaries, and skip_word_none is set to TRUE, then it returns an empty character vector. For non-word boundaries, this fix sets skip_word_none to FALSE unless the user has deliberately chosen otherwise.

str_split("This is a sentence. So is this.", boundary("sentence"))
#> [[1]]
#> [1] "This is a sentence. " "So is this."

The PR adds tests for sentence splitting.

[gagolews]

@lmullen that's right.

[hadley]

Maybe it would be better to make the default value of skip_word_none NA, and then do:

if (identical(skip_word_none, NA)) {
  skip_word_none <- type == "word"
}

This would also need some doc updates

[dselivanov]

when this will be merged?

[hadley]

@lmullen do you want to finish this off? It also needs a bullet point in NEWS

[lmullen]

@hadley Sorry, I screwed up squashing the pull request. Mind if I resubmit this as a new, clean PR?

[hadley]

Yeah, sure