Redesigning tokens_segment()

[koheiw]

I hate to say, but we have to revisit tokens_segment (and probably corpus_segment), because tokens segmentation seems really important to solve two important issues (#516 and #521).

Negations #516

dict <- data_dictionary_LSD2015[1:2]

toks <- tokens("The share of the car is not good, but the color is nice.") # neutral sentence
mx <- dfm(toks, dictionary = dict)

# select 3 word-window after 'not'
toks_negation <- as.tokens(list(as.list(toks)[[1]][7:9])) 
# toks_negation <- tokens_segment(toks, 'not', window_after = 3) # proposed approach
mx_negation <- dfm(toks_negation, dictionary = dict)

as.vector(mx[,2] - mx[,1]) # 2 (incorrect)
as.vector(mx[,2] - mx[,1]) - (as.vector(mx_negation[,2] - mx_negation[,1]) * 2) # 0 (correct)

Target-word based collocation #521

# select 2 word-window before and after 'car'
toks_target <- as.tokens(list(as.list(toks)[[1]][3:7]))
# toks_negation <- tokens_segment(toks, 'not', window = 2) # proposed approach
mx_target <- dfm(toks_target)

textstat_keyness(rbind(dfm(toks_target), dfm(toks)))

# chi2         p n_target n_reference
# is     0.1307190 0.7176878        1           2
# of     0.0000000 1.0000000        1           1
# the    0.0000000 1.0000000        1           3
# car    0.0000000 1.0000000        1           1
# not    0.0000000 1.0000000        1           1
# share -0.3508772 0.5536170        0           1
# good  -0.3508772 0.5536170        0           1
# ,     -0.3508772 0.5536170        0           1
# but   -0.3508772 0.5536170        0           1
# color -0.3508772 0.5536170        0           1
# nice  -0.3508772 0.5536170        0           1
# .     -0.3508772 0.5536170        0           1

We can add this function in a different name from tokens_segment(), but the name seems the most appropriate. Current tokens_segment() should be called tokens_segment_tag() as tag extraction is a special case of segmentation.

Alternatively, we could treat this operations as selection of tokens, in stead of segmentation, and add a function called tokens_select_window().

[kbenoit]

I see the problems, not sure what the best approach is yet to the solution. tokens_segment(), like corpus_segment(), finds elements inside the documents to split and redefine the document units, producing a new tokens or corpus object following the split. We should keep it that way.

On the target-word based collocation

To tokens_segment(), yes we could add a window argument, through modifying pattern_position to include middle. But we almost already have the needed functionality through kwic() (we just need to include the keyword and not split on pre and post):

kwic(toks, "not", 3) %>% corpus() %>% dfm()
Document-feature matrix of: 2 documents, 6 features (50% sparse).
# 2 x 6 sparse Matrix of class "dfmSparse"
#             features
# docs         the car is good , but
#   text1.pre    1   1  1    0 0   0
#   text1.post   0   0  0    1 1   1

Suggest we keep this a separate issue though.

On negations

This looks like a bug:

tokens_lookup(toks, dictionary = data_dictionary_LSD2015, exclusive = TRUE)
# tokens from 1 document.
# text1 :
# [1] "neg_positive" "positive"     "positive"    

tokens_lookup(toks, dictionary = data_dictionary_LSD2015, exclusive = FALSE)
tokens from 1 document.
# text1 :
#  [1] "NEG_NEGATIVE" "The"          "share"        "of"           "the"         
#  [6] "car"          "NEG_POSITIVE" "POSITIVE"     "good"         ","           
# [11] "of"           "but"          "car"          "POSITIVE"     "nice"   

Putting aside the odd NEG_NEGATIVE for now, we should make a decision about nested matches to that we don't count "not good" as both NEG_POSITIVE and POSITIVE. This is the same issue as we discussed in #970.

[kbenoit]

For the broader matches of context for our dictionaries, including the ability to cross-reference other keys, we should consider allowing rules such as what Wordstat has - see pp69- of the WordStat manual (Working with [Dictionary] Rules).

This goes all the way back to #516.

[koheiw]

I want a separate function from tokens_segment(), because it does too many things to only to extract matched patters, which is not always necessary. Although I put the title "redesigning tokens_segment" (with typo), I favor the tokens_select_window() option, because it is much more efficient.

kwic() is also an inefficient function, because it create character vectors from serialized tokens.

[koheiw]

As for negation, it is really difficult to implement NEAR operator (as in WordStat) in the current architecture, so better to find a different way for now (until v2.0).