kwic behaviour needs updating for phrase() scheme

[kbenoit]

I propose that we only match sequences of keywords if wrapped in phrase(). (Once PR #820 is merged) see the tests in test-kwic.R that need activation for CI. If these pass, then my proposed behaviours will have been implemented.

We can use this issue for discussing them, and implement the changes in a separate PR.

[koheiw]

There are two separate issues here: interpretation of keyword input, and presentation of concordance.

After having changed how kwic() interprets the input, "i j" does not match, but why there is 'a b'? This is because kwic() presents adjacent matches in a single line in a way similar to join = TRUE in tokens_compound().

txt <- c(d1 = "a b c d e g h",  d2 = "a b e g h i j")
char_uni <- c("a", "b", "g", "j", "i j")
toks_uni <- tokens(txt)
kwic(txt, char_uni)

# [d1, 1:2]           | a b | c d e g h
# [d1, 6:6] a b c d e |  g  | h        
# [d2, 1:2]           | a b | e g h i j
# [d2, 4:4]     a b e |  g  | h i j    
# [d2, 7:7] b e g h i |  j  |    

It could be like this, but less compact:

# [d1, 1:1]           | a | b c d e g
# [d1, 2:2]         a | b | c d e g h
# [d1, 6:6] a b c d e | g | h        
# [d2, 1:1]           | a | e g h i j
# [d2, 2:2]         a | b | e g h i j
# [d2, 4:4]     a b e | g | h i j    
# [d2, 7:7] b e g h i | j |   

[kbenoit]

I strongly prefer the latter as the default, since it means we only return the keyword that is matched, not sequences of separate keywords. If someone wants to get the keyword "a b", they should use keyword = phrase("a b") as the keyword.

[koheiw]

We can make the latter the default, but I need the former. For example, when I count words that appear near 'a' or 'b', 'c', 'd', 'e' are double counted in the letter. Solution is to add join argument to kwic() or make a separate function like tokens_context() as I propose earlier.

[kbenoit]

Join would be consistent with tokens_compound and preferable to adding a new function. Let's go with that and set the default to FALSE.

[kbenoit]

Perfect. Have you tested it with the phrase wrapper?