tokens_select not selecting ngrams

[kbenoit]

I thought we had addressed this already, but maybe this is part of #719.

for tokens

Define two sets of tokens, simple unigrams and space-separated bigrams:

(toks <- tokens("a b c d e f g"))
# tokens from 1 document.
# Component 1 :
# [1] "a" "b" "c" "d" "e" "f" "g"

(toks2 <- tokens_ngrams(toks, n = 2, concatenator = " "))
# tokens from 1 document.
# Component 1 :
# [1] "a b" "b c" "c d" "d e" "e f" "f g"

Then the first works, but the second fails. I'd like the second to work the same as the first.

tokens_select(toks, c("a", "e", "g"))
# tokens from 1 document.
# Component 1 :
# [1] "a" "e" "g"

tokens_select(toks2, c("a b", "d e", "f g"))
# tokens from 1 document.
# Component 1 :
# character(0)

for dfms

The behaviour is correct for dfms already formed, but incorrect when forming, because I think tokens_select() is called before creating the dfm and then using dfm_select() inside dfm().

dfm(toks2, select = c("a b", "d e", "f g"))
# NULL

dfm_select(dfm(toks2), c("a b", "d e", "f g"))
# Document-feature matrix of: 1 document, 3 features (0% sparse).
# 1 x 3 sparse Matrix of class "dfmSparse"
#        features
# docs    a b d e f g
#   text1   1   1   1

[koheiw]

You have to do like this to select ngrams separated by spaces:

tokens_select(toks2, list("a b", "d e", "f g"))

if the input is a vector, they are multi-word features.

[kbenoit]

Here's an example of awkward syntax for using this structure: http://stackoverflow.com/questions/44030170/compute-chi-square-value-between-ngrams-and-documents-with-quanteda/.

Too complicated?

[koheiw]

You do not need to make a list. Just do like this:

corp_example <- data_corpus_inaugural
corp_example <- corpus_subset(corp_example, Year > 1960)

# this will remove punctuation and numbers
toks_example <- tokens(corp_example, remove_punct = TRUE, remove_numbers = TRUE)

# find and score chi^2 bigrams
coll2 <- textstat_collocations(toks_example, method = "chi2", max_size = 2)
head(coll2, 10)

# collocations (faster)
toks_example_comp <- tokens_compound(toks_example, coll2[1:100])
toks_example_comp[[1]][1:20]

# character vector
toks_example_comp2 <- tokens_compound(toks_example, coll2$collocation[1:100])
toks_example_comp2[[1]][1:20]

[koheiw]

More tricky case,

# ngrams are joined with whitespaces
toks_example_comp3 <- tokens_compound(toks_example, coll2[1:100], concatenator = ' ')
toks_example_comp3[[1]][1:20]

# features must be a list
toks_example_comp4 <- tokens_select(toks_example_comp3, as.list(coll2$collocation[1:100]))
toks_example_comp4[[1]][1:20]

I just say to users that do not join ngrams with whitespace unless you know quanteda very well.

[kbenoit]

The original example still does not work:

toks2
# tokens from 1 document.
# Component 1 :
# [1] "a b" "b c" "c d" "d e" "e f" "f g"
 
tokens_select(toks2, c("a b", "d e", "f g"))
# tokens from 1 document.
# Component 1 :
# character(0)

I don't think it ought be a list for the features, since it's not for dfm_select(). Or we stick more closely to our definitions, that tokens are not features, only dfms have features, and define those differently. This is part of the broader issue in #719.

[koheiw]

Please have a look at #804 and comment on features.R. Otherwise I cannot do anything with this.

[kbenoit]

will do. I will think hard about this and issue a (comprehensive) proposal for comment.