Repository navigation
check calculations of textstat_collocations() #803
Description
Here's a stab at a log-linear model equivalency. The focus is on "capital gains tax".
require(quanteda)
txt <- "A gains capital B C capital gains A B capital C capital gains tax gains tax gains B gains C capital gains tax"
toks <- as.character(tokens(txt, ngrams = 3, concatenator = " "))
require(data.table)
toks_df <- data.table(do.call(rbind, strsplit(toks, " ")), stringsAsFactors = FALSE)
names(toks_df) <- paste0("word", 1:3)
# replace non-target words with "other"
targets <- c("capital", "gains", "tax")
toks_df[word1 != targets[1], word1 := "other"]
toks_df[word2 != targets[2], word2 := "other"]
toks_df[word3 != targets[3], word3 := "other"]
toks_df[, n := 1]
toks_df_n <- toks_df[, list(n = sum(n)), by = c("word1", "word2", "word3")]
toks_df_n
# word1 word2 word3 n
# 1: other gains other 3
# 2: other other other 12
# 3: capital other other 2
# 4: capital gains other 1
# 5: capital gains tax 2
# 6: other gains tax 1
# MASS::loglm(n ~ word1:word2:word3, data = toks_df)
stats::loglin(table(toks_df[, 1:3]), margin = 1:3)
# 2 iterations: deviation 1.776357e-15
# $lrt
# [1] 10.91587
#
# $pearson
# [1] 16.93125
#
# $df
# [1] 4
#
# $margin
# [1] "word1" NA NA
textstat_collocations(tokens(txt), method = "lr", min_size = 3, max_size = 3)[1:3, ]
# collocation length count G2
# 1 C capital gains 3 3 20.17961
# 2 gains tax gains 3 2 11.18707
# 3 capital gains tax 3 2 10.91587
textstat_collocations(tokens(txt), method = "chi2", min_size = 3, max_size = 3)[1:3, ]
# collocation length count X2
# 1 C capital gains 3 3 43.95563
# 2 gains tax gains 3 2 23.64158
# 3 capital gains tax 3 2 16.93125
Activity
HaiyanLW commentedon Jun 15, 2017
For the coefficient of
Dice, McInnes uses2for trigrams and Petrovic uses3(sonforn-grams collocation). Currently we adopted Petrovic's way insequences.Rand McInnes's way incollocations2.R.kbenoit commentedon Jun 15, 2017
OK - It turns out that we were computing the model that included lower order interactions, which means that the likelihood ratio we are computing for ngrams with n>2 are for the wrong model. What we should be computing (for trigrams) is the likelihood of seeing the third word after having seen the first two words in order.
In contrast to the above, this has the following equivalent formulations in log-linear models:
As a binary logistic regression, what we are computing is the coefficient for the interaction term when ther data are expressed this way:
Since Blaheta and Johnson smooth their counts, I've tried to approximate their model:
kbenoit commentedon Jun 15, 2017
From
sequences():HaiyanLW commentedon Jun 19, 2017
According to the
2x2table inslackon last Friday:My understanding is as the following:
and from the updated
sequences()now,HaiyanLW commentedon Jun 19, 2017
kbenoit commentedon Jun 19, 2017
That's it. There are faster algebraic methods to compute all of these, incl lambda and sigma, without the iterative approach in
stats::loglin(). See my code intextstat_keyness()for instance for the quick computations of the chi^2 and G^2 for the 2x2 tables. (They might be using the Yates correction of adding 0.5 to each cell...)It's all so much easier with 2x2 tables!
HaiyanLW commentedon Jun 19, 2017
Do you think we should move the calculation of the chi^2 and G^2 (and maybe pmi) outside c++ code? I mean these are computed paralleled now. And once we move it to R, we still need to calculate them separately for 2-, 3- ...n-grams.
kbenoit commentedon Jun 19, 2017
If the calculation is vectorised in R, it will be as fast as in C++, and easier to maintain within the package, so I'm in favour. This would mean we are just using the C++ to form the matches and return the tables.
HaiyanLW commentedon Jun 19, 2017
Now the
chi^2,G^2,pmiandDiceare returned withlambdaandlambda1, and are tested againstloglin()for tri-grams, shall we leave it as such until we decide how to updatetextstat_collocations.R?kbenoit commentedon Jun 19, 2017
Fine with me.
HaiyanLW commentedon Jun 19, 2017
:) seems need a break from this branch.
HaiyanLW commentedon Jun 19, 2017
Do you want me do a PR or leave it for now?
kbenoit commentedon Jun 19, 2017
If the tests are there to verify the calculations, we use them to test other implementations, such as the one that @koheiw is working on). So if it's complete, go ahead and issue a PR. Merge the latest
masterchanges into the branch first, if you have not yet done that.Merge pull request #809 from kbenoit/hw-collocations
kbenoit commentedon Aug 2, 2017
@HaiyanLW here some some code that computes, using a slow R way, the expected values. The figures are taken from Jouni's example, and I have not implemented smoothing, but I think this will be already a part of the "observed" values that I have computed, so it should work as is. If you can integrate this temporarily into the
collocations_verifybranchtextstat_collocations(), we can use that to verify the figures.HaiyanLW commentedon Aug 2, 2017
For now (specify size to scalar and only 2 or 3), hasn't changed
G2etc for 3-, 4-, 5-grams:kbenoit commentedon Aug 2, 2017
OK when you push it to the branch I will run some tests. Please merge
masterinto your branch first.