dfm_lookup() and tokens_lookup() disagree

[koheiw]

Looks like a bug. "GB" in "text2" is 2 and 3.

require(quanteda)
#> Loading required package: quanteda
#> Warning: package 'quanteda' was built under R version 4.0.5
#> Warning in stringi::stri_info(): Your native charset is not a superset of US-
#> ASCII. This may cause serious problems. Consider switching to UTF-8.

#> Warning in stringi::stri_info(): Your native charset is not a superset of US-
#> ASCII. This may cause serious problems. Consider switching to UTF-8.
#> Package version: 3.2.0
#> Unicode version: 13.0
#> ICU version: 69.1
#> Parallel computing: 8 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
require(newsmap)
#> Loading required package: newsmap
#> Warning: package 'newsmap' was built under R version 4.0.5
txt <- c("American and Japanese leaders met in Tokyo.",
         "Paris Hilton visited British museum in London.",
         "India and Pakistan are neighbours.",
         "A man went to the Moon.")
toks <- tokens(txt)
toks_label <- tokens_lookup(toks, data_dictionary_newsmap_en, levels = 3)
dfmt <- dfm(toks)
dfmt_label <- dfm(toks_label, tolower = FALSE)
dfmt_label[,c("GB", "JP", "FR")]
#> Document-feature matrix of: 4 documents, 3 features (75.00% sparse) and 0 docvars.
#>        features
#> docs    GB JP FR
#>   text1  0  2  0
#>   text2  2  0  1
#>   text3  0  0  0
#>   text4  0  0  0

dfmt_label2 <- dfm_lookup(dfmt, data_dictionary_newsmap_en, levels = 3)
dfmt_label2[,c("GB", "JP", "FR")]
#> Document-feature matrix of: 4 documents, 3 features (75.00% sparse) and 0 docvars.
#>        features
#> docs    GB JP FR
#>   text1  0  2  0
#>   text2  3  0  1
#>   text3  0  0  0
#>   text4  0  0  0

^{Created on 2022-01-24 by the reprex package (v2.0.1)}

[kbenoit]

Looks like a bug in dfm_lookup(). We didn't implement the same type of nesting behaviours as in tokens_lookup(). So it appears there is a double match on "British", because:

> data_dictionary_newsmap_en$EUROPE$NORTH$GB
[1] "uk"             "united kingdom" "britain"        "british"        "briton*"        "brit*"         
[7] "London"

Minimally reproducible example:

library("quanteda")
## Package version: 3.2.0.9000
## Unicode version: 13.0
## ICU version: 69.1
## Parallel computing: 12 of 12 threads used.
## See https://quanteda.io for tutorials and examples.

dict <- dictionary(list(GB = c("brit*", "british")))
toks <- tokens("British Museum")
tokens_lookup(toks, dict)
## Tokens consisting of 1 document.
## text1 :
## [1] "GB"

dfm(toks) %>%
  dfm_lookup(dict)
## Document-feature matrix of: 1 document, 1 feature (0.00% sparse) and 0 docvars.
##        features
## docs    GB
##   text1  2

[koheiw]

You are right. This is the nesting issue. I will try to fix.