dfm_lookup needs "non-matched" column option

[kbenoit]

For computing relative frequencies in a dfm after applying a dictionary, what is usually needed is the dictionary count relative to all features. But dfm_lookup only keeps features found. To get relative frequencies of all tokens requires some workarounds, but it would be better to have directly. (An earlier version of dfm() did behave this way.)

require(magrittr)

txt <- c(d1 = "a b c a a b d",
         d2 = "a a b d", 
         d3 = "a b d D d D d")
toks <- tokens(txt)

mydict <- dictionary(list(A = "a", BC = c("b", "c")))

# how to get frequencies of non-dictionary items?
dfm(toks, tolower = FALSE, dictionary = mydict) %>% 
    dfm_weight("relFreq")
## Document-feature matrix of: 3 documents, 2 features (0% sparse).
## 3 x 2 sparse Matrix of class "dfmSparse"
##     features
## docs         A        BC
##   d1 0.5000000 0.5000000
##   d2 0.6666667 0.3333333
##   d3 0.5000000 0.5000000

# not idea since cannot distinguish keys from original features
dfm(toks, tolower = FALSE) %>% 
    dfm_lookup(dictionary = mydict, exclusive = FALSE) %>% 
    dfm_weight("relFreq")
## applying a dictionary consisting of 2 keys
## Document-feature matrix of: 3 documents, 4 features (16.7% sparse).
## 3 x 4 sparse Matrix of class "dfmSparse"
##            d         D         A        BC
## d1 0.1428571 0         0.4285714 0.4285714
## d2 0.2500000 0         0.5000000 0.2500000
## d3 0.4285714 0.2857143 0.1428571 0.1428571



# workaround
tokens_select(toks, unlist(mydict), padding = TRUE) %>%
    tokens_lookup(dictionary = mydict, exclusive = FALSE) %>%
    dfm(tolower = FALSE) %>%
    dfm_weight("relFreq")
## Document-feature matrix of: 3 documents, 3 features (0% sparse).
## 3 x 3 sparse Matrix of class "dfmSparse"
##     features
## docs                   A        BC
##   d1 0.1428571 0.4285714 0.4285714
##   d2 0.2500000 0.5000000 0.2500000
##   d3 0.7142857 0.1428571 0.1428571

[koheiw]

I think this is related to #479.

[kbenoit]

I think this is related to #479.

Only partially - my issue is not to pad non-existent features with zeroes, but rather to count all tokens not matching a dictionary key as an "Other" count. The workaround does this by counting the pads "" after removing all matches to keys, then applying the dictionary to the tokens, then passing to straight dfm() which preserves the pads as a "" feature.

[koheiw]

How about recording the lengths() of tokens in an attribute before the exclusive=TRUE tokens lookup, and use it in dfm_weight()?

[kbenoit]

Excellent idea that would work for dfm_lookup too.