Nested (overlapped) elements to tokens_lookup()

[R01010010R]

tokens_lookup() returns more than value if a dictionary has overlapping elements (see koheiw/newsmap#13 (comment)).

There is a workaround which works.

1. Use tokens_lookup with the option valuetype = 'glob'.
2. Check if the tokens also return at least one of the values when I use tokens_lookup with the option valuetype = 'fixed'. And then drop the other values from (1).

However, an option such as the one @koheiw suggested might be more efficient and easier to implement.

[plugrafico]

Can dictionaries by assembled so that categories follow logical commands to ignore hits by neighboring categories?
For instance (just exemplifying, not really sure how the logical expressions should look like):

dict <- dictionary(list(
guinea_bissau="guinea bissau", 
equatorial_guinea="equatorial guinea",
guinea="guinea(NOT 'guinea_bissau' NOT 'equatorial_guinea')")

[kbenoit]

Really should be a SO question, but anyway:

The easiest way would be to compound the tokens you want treated as single items. You can also look up sequences and match them using dictionaries.

library("quanteda")
## Package version: 1.4.3

txt <- c("I spent one guinea while visiting Equatorial Guinea and then went to Guinea Bissau")
toks <- tokens(txt)

# method 1: compound the tokens first
tokens_compound(toks, phrase(c("equatorial guinea", "guinea bissau")))
## tokens from 1 document.
## text1 :
##  [1] "I"                 "spent"             "one"              
##  [4] "guinea"            "while"             "visiting"         
##  [7] "Equatorial_Guinea" "and"               "then"             
## [10] "went"              "to"                "Guinea_Bissau"

# method 2: define just the compound word dictionaries
dict <- dictionary(list(
  Equatorial_Guinea = "equatorial guinea",
  Guinea_Bissau = "guinea bissau"
))
tokens_lookup(toks, dict, exclusive = FALSE)
## tokens from 1 document.
## text1 :
##  [1] "I"                 "spent"             "one"              
##  [4] "guinea"            "while"             "visiting"         
##  [7] "EQUATORIAL_GUINEA" "and"               "then"             
## [10] "went"              "to"                "GUINEA_BISSAU"

Set capkeys = FALSE to turn off the capitalization.

[plugrafico]

Thanks for clearing that, @kbenoit

[kbenoit]

No problem and @koheiw and I are working on more built-in solutions too (#1708).