Multi-word entries in dictionary appear to be ignored

[pablobarbera]

The following code is a simplified version of the example in quanteda::dictionary

mycorpus <- subset(inaugCorpus, Year>1900)
mydict <- dictionary(list(country = "united states"))
sum(dfm(mycorpus, dictionary = mydict)[,"country"])

Another example:

mycorpus <- corpus("this should work")
mydict <- dictionary(list(example = "should work"))
sum(dfm(mycorpus, dictionary = mydict)[,"example"])

Trying with "_" as concatenator:

mycorpus <- corpus("this should work")
mydict <- dictionary(list(example = "should_work"), concatenator="_")
sum(dfm(mycorpus, dictionary = mydict)[,"example"])

Am I missing something? This came up as I was trying to use quanteda with Lexicoder, which has multi-word entries in the dictionary. I'm running quanteda 0.9.6-9

┆Issue is synchronized with this Asana task

[conjugateprior]

If you've got Java 8 installed, which I guess you might if you're using lexicoder, then you could use jca while this gets fixed.

[kbenoit]

@pablobarbera This feature is not live yet, but there are workarounds using "phrasetotoken". But lots of demand for this exists so we need to add it asap.

@koheiw let's talk about how the newest selectFeatures mods could be used for this purpose, prior to dfm construction.

[koheiw]

Joining tokens before applying dfm is the way. It is not automated, but I am sure that you can extract multi-part words easily from the dictionary.

seqs <- list(c('should', 'work'), c('united', 'states'))
toks2 <- joinTokens(toks, seqs, "_")

[kbenoit]

@koheiw I was working today on a foundation to get this issue resolved, in the dev_multiWordDictionaries branch. Please see the changes I made to joinTokens().

1. I will eventually fold this function into phrasetotoken with a signature for tokenizedTexts, vector/list of phrases. The function I added for this (bottom of phrases.R) currently calls joinTokens().
2. The examples in the roxygen2 header show some cases where it's not working. There is a bizarre outcome based on which match is first, probably because of the loop's sequential processing of the token sequences. Can you investigate this? (See the @examples)
3. We need a case_insensitive argument, which I added, but it does not work in your C++ code that performs the substitutions. Can you look at the code and think about how to make this work? It translates nicely to the stringi match functions but does not work in your code.
4. Note that I removed the code to a new function, regexToFixed(), that converts the regex patterns to the fixed elements found in the set of types, so that the fixed matching can be used. I took it out of joinTokens() so that other code could use it too.

Please work directly on the dev_multiWordDictionaries branch.

[koheiw]

I fixed many of the problems. The key is the way to use the code now in grid_sequence (regexToFixed). joinTokens() has to generate all possible patterns for case-insensitive concatenation. There were some issues in C++ as well, but I fixed them.

[conjugateprior]

Is there a written statement of the semantics of multiword pattern matching someplace? I'd be happy to help on this branch, but my experience from jca is that a) this is trickier than it looks, and b) treating multiword matches as single tokens makes various marginal statistics subtly inconsistent.

[Seneca75]

Hello!
Any idea when this problem might be solved? I would like to use multiword dictionaries in text analyses. Joining tokens does not seem to be the right way here because of various issues (e.g. tampering with the word counts).

[kbenoit]

Not yet, but I'd say by mid-September!

[Seneca75]

Great news! Thanks a lot!

[kbenoit]

I added new functionality to implement dictionaries for with multi-word keys, new method applyDictionary.tokenizedTexts(). In master as of v.0.9.8.8. See the examples in test_dictionary_multiword.R.
From commit d76972a (but includes loads of updates from merging in the master, since that branch had gotten quite stale).

It includes a double match for a spots and a country category for the phrase "Manchester United States" - @conjugateprior is addressing this sort of issue what you meant by defining the semantics of multi-word matching?

Still need to implement pattern matches, as well as a method for the hashed tokenizedTexts object class, but these will not be difficult.

@koheiw Happy to compare speeds with the C++ versions - what I wrote today in R is pretty smokin' fast and in fact, much faster even for single-word dictionary keys than creating a dfm and then applying the dictionary.