Skip to content

dfm_select fails when features are ngrams #589

Description

@kbenoit

This works:

dfm1 <- dfm(data_corpus_inaugural, ngrams = 1, removePunct = TRUE)
(topf1 <- topfeatures(dfm1))
##   the    of   and    to    in     a   our  that    we    be 
## 10082  7103  5310  4526  2785  2246  2181  1789  1739  1481 
head(dfm_select(dfm1, names(topf1)))
# Document-feature matrix of: 58 documents, 10 features (0.517% sparse).
# (showing first 6 documents and first 6 features)
#                  features
# docs               of the and to that  a
#   1789-Washington  71 116  48 48   18 14
#   1793-Washington  11  13   2  5    1  0
#   1797-Adams      140 163 130 72   22 51
#   1801-Jefferson  104 130  81 61   24 21
#   1805-Jefferson  101 143  93 83   37 20
#   1809-Madison     69 104  43 61    9 19

but this fails:

dfm2 <- dfm(data_corpus_inaugural, ngrams = 2, removePunct = TRUE)
(topf2 <- topfeatures(dfm2))
## of_the     in_the     to_the     of_our    and_the      it_is     by_the    for_the      to_be the_people 
##   1766        812        720        619        471        323        318        312        310        265 
head(dfm_select(dfm2, names(topf2)))
## NULL

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions