collocations question

[Jim89]

Hi Ken,

I have a question about quanteda collocations that I'm hoping you might be able to help with.

I'm looking for collocations of terms within a corpus of customer complaints and I've turned again to quanteda to help with this.

I notice that when you calculate the collocations your first tokenise the texts with simplify = TRUE, returning a simple vector of all tokens across all documents in the corpuse rather than tokens within a document.

This means that I'm detecting collocations that occur across documents in the corpus, not just within them and I'm not sure if that's correct.

For example if document 1 contains the text "This is a test" and document 2 contains the text "This is also a test" then the collocation "test this" is returned, even though it doesn't exist within a document in the corpus.

Are you able to shed any light on why this is the behaviour of collocations, and if this is correct?

Thanks very much and I hope you're well. I'm enjoying the work I'm doing with analysis of text and quanteda makes things a lot easier!

Jim.

[adamobeng]

I can confirm that this behaviour occurs even with a tokenizedTexts argument and size=2, which doesn't cause the argument to be tokenized with simplify=FALSE.

> collocations(
+   tokenize(c('this is a test', 'this is also a test'), simplify=T)
+ )
   word1 word2 word3 count        G2
1:     a  test           2 8.9973623
2:  this    is           2 8.9973623
3:  test  this           1 6.0283226
4:  also     a           1 3.2557339
5:    is  also           1 3.2557339
6:    is     a           1 0.8180391
> collocations(
+   tokenize(c('this is a test', 'this is also a test'))
+ )
   word1 word2 word3 count        G2
1:     a  test           2 8.9973623
2:  this    is           2 8.9973623
3:  test  this           1 6.0283226
4:  also     a           1 3.2557339
5:    is  also           1 3.2557339
6:    is     a           1 0.8180391
> collocations(
+   tokenize(c('this is a test.', 'this is also a test'))
+ )
   word1 word2 word3 count        G2
1:     a  test           2 8.3757742
2:  this    is           2 8.3757742
3:  also     a           1 2.9690397
4:    is  also           1 2.9690397
5:    is     a           1 0.5991613

Note that collocations don't span punctuation by default, however...

[kbenoit]

Hi @Jim89, glad you're using the package!

Behaviour looks as intended to me, but don't remove the punctuation from the test examples!

> collocations(tokenize(c("This is a test.", "This is also a test."), simplify = TRUE))
   word1 word2 word3 count        G2
1:     a  test           2 8.3757742
2:  this    is           2 8.3757742
3:  also     a           1 2.9690397
4:    is  also           1 2.9690397
5:    is     a           1 0.5991613
> collocations(tokenize(c("This is a test.", "This is also a test."), simplify = TRUE), 
               spanPunct = TRUE)
   word1 word2 word3 count        G2
1:     a  test           2 10.008048
2:  test     .           2 10.008048
3:  this    is           2 10.008048
4:     .  this           1  6.501659
5:  also     a           1  3.729071
6:    is  also           1  3.729071
7:    is     a           1  1.207137

See the spanPunct argument in collocations(). And whatever you pass in ... in collocations(x, ...) is sent to tokenize(). Except when you don't use ... ... maybe this behaviour is counter-intuitive.

[adamobeng]

@kbenoit But what if the texts in the corpus don't in fact end with punctuation?

[Jim89]

Alright, thanks for the speedy response both. Most of the texts I'm working with don't end with punctuation, so I can add a full stop as a work around. Looks like @adamobeng asked the question I was about to ask just ahead of me, though. Thanks! 😃

[adamobeng]

Without detracting from the original question, does it make sense that spanPunct=TRUE returns collocations that include punctuation? I would have expected the punctuation to be stripped, but tokens on either side of punctuation to be considered collocated.

[kbenoit]

Some interesting questions here, involving behavioural decisions. collocations() is designed to work, like most quanteda functions, on a corpus, character, or tokenizedTexts object. The idea is to make this easy - since most people will just want to mine a corpus for collocations - but also to give a power user control over the process if he/she wants to start with texts manually tokenised. Currently, the ... is passed through to tokenize()to allow this control even when starting from a corpus or character input.

Current behaviour, for character:

> char <- "This, a test."
> collocations(char)
   word1 word2 word3 count            G2
1:     a  test           1 -2.220446e-16
> collocations(char, spanPunct = FALSE)
   word1 word2 word3 count            G2
1:     a  test           1 -2.220446e-16
> collocations(char, spanPunct = TRUE)
   word1 word2 word3 count       G2
1:     ,     a           1 5.004024
2:     a  test           1 5.004024
3:  test     .           1 5.004024
4:  this     ,           1 5.004024
> collocations(char, spanPunct = FALSE, removePunct = TRUE)
   word1 word2 word3 count       G2
1:     a  test           1 3.819085
2:  this     a           1 3.819085
> collocations(char, spanPunct = TRUE, removePunct = TRUE)
   word1 word2 word3 count       G2
1:     a  test           1 3.819085
2:  this     a           1 3.819085

For a tokenizedTexts object:

> collocations(tokenize(char))
   word1 word2 word3 count            G2
1:     a  test           1 -2.220446e-16
> collocations(tokenize(char), spanPunct = FALSE)
   word1 word2 word3 count            G2
1:     a  test           1 -2.220446e-16
> collocations(tokenize(char), spanPunct = TRUE)
   word1 word2 word3 count       G2
1:     ,     a           1 5.004024
2:  This     ,           1 5.004024
3:     a  test           1 5.004024
4:  test     .           1 5.004024
> collocations(tokenize(char, removePunct = TRUE), spanPunct = FALSE)
   word1 word2 word3 count       G2
1:  This     a           1 3.819085
2:     a  test           1 3.819085
> collocations(tokenize(char, removePunct = TRUE), spanPunct = TRUE)
   word1 word2 word3 count       G2
1:  This     a           1 3.819085
2:     a  test           1 3.819085

input	spanPunct	removePunct	result
char			a test
char	FALSE		a test
char	TRUE		c(", a ", "a test", "test .", "this ,")
char	FALSE	TRUE	c("a test", ""this a")
char	TRUE	TRUE	c("a test", ""this a")
toks			a test
toks	FALSE		a test
toks	TRUE		c(", a ", "a test", "test .", "this ,")
char	FALSE	TRUE	c("This a", "a test")
char	TRUE	TRUE	c("This a", "a test")

[adamobeng]

I'm not sure I quite understand.

Here is the current behaviour in a few different situations:

number	call	result
1	collocations(c('A B', 'C D'))	(a, b), (b, c), (c, d)
2	collocations(c('A B.', 'C D'))	(a, b), (c, d)

@Jim89 thinks 1 should be (a, b), (c, d), and I agree.

Separately:

number	call	result
3	collocations(c('A B. C D'), spanPunct=T)	(., c), (a, b), (b, .), (c, d)
4	collocations(c('A B. C D'), removePunct=T)	(a, b), (b, c), (c, d)
5	collocations(c('A B. C D'), removePunct=T, spanPunct=T)	(a, b), (b, c), (c, d)

I also think 3 should be (a, b), (b, c), (c, d), or at least (., c), (a, b), (b, c), (b, .), (c, d).

[Jim89]

I agree with what @adamobeng said.

Regardless of whether or not the documents in the corpus contain an ending punctuation mark, collocations of ('A B', 'C D') should in my head return (a, b) and (c, d). The current behaviour that generates (b, c) as well can end up with misleading results.

It's something of an edge case, but it's one that I just found to be important "in the wild", especially when dealing with very short documents (I'm working with comments of ~10 words).

@kbenoit, I think your recent commit addresses this nicely, thanks for such a quick turnaround!

[Jim89s]

@kbenoit Additionally, is it possible to push the changes to collocations to CRAN, please? I'd like to use them in our work but I can't create work products that are reliant on non-CRAN R packages.

[kbenoit]

Sure - we will work on this asap.

[Jim89s]

Brill. Thanks @kbenoit, it'll save me some horrible hack workarounds!