Repository navigation
Closed
Description
(No doubt this is also related to #306.)
require(quanteda)
## Loading required package: quanteda
## quanteda version 0.9.8.12
mycorpus <- corpus(c("This is a test",
"This is it.",
"What is in a train?",
"Is it a question?",
"Sometimes you don't know if this is it.",
"Is it a bird or a plane or is it a train?"))
# CORRECT FAIL: because this is parsed as three tokens, with no match
# for the middle token, "is|sometimes"
kwic(mycorpus, keywords = c("this is|sometimes you"), valuetype = "regex")
## [1] NA
# INCORRECT FAIL: should also match text2
kwic(mycorpus, keywords = c("is it|in a"), valuetype = "regex")
## [text4, 1:3] [ Is it a ] question?
## [text6, 1:3] [ Is it a ] bird or a plane or
## [text6, 9:11] bird or a plane or [ is it a ] train?
# INCORRECT FAIL: should also match text3
kwic(mycorpus, keywords = c("is it|in a"), valuetype = "regex")
## [text4, 1:3] [ Is it a ] question?
## [text6, 1:3] [ Is it a ] bird or a plane or
## [text6, 9:11] bird or a plane or [ is it a ] train?
# INCORRECT FAIL: should also match text3
kwic(mycorpus, keywords = c("is i? a"), valuetype = "glob")
## [text4, 1:3] [ Is it a ] question?
## [text6, 1:3] [ Is it a ] bird or a plane or
## [text6, 9:11] bird or a plane or [ is it a ] train?
# INCORRECT FAIL: Individual matches should work in a vector too
# --- this is ok:
kwic(mycorpus, keywords = c("is it"), valuetype = "regex")
## contextPre keyword contextPost
## [text2, 2:3] This [ is it ]
## [text3, 2:3] What [ is it ]
## [text4, 1:2] [ Is it ] a question?
## [text5, 7:8] you don't know if this [ is it ]
## [text6, 1:2] [ Is it ] a bird or a plane
## [text6, 9:10] bird or a plane or [ is it ] a train?
# --- this is ok:
kwic(mycorpus, keywords = c("or a"), valuetype = "regex")
## contextPre keyword contextPost
## [text6, 5:6] Is it a bird [ or a ] plane or is it a
# -- NOT OK:
kwic(mycorpus, keywords = c("is it", "or a"), valuetype = "regex")
## [1] NAThanks to @jjphorn for bringing this to my attention.
Activity
Potential fix for #291 #306 #307
koheiw commentedon Jan 2, 2017
I do not think we should allow use of
|operator.keywords = c("is it|in a")should bekeywords = c("is it"), ("in a"), or more strictlykeywords = list(c("is", "it"), c("in", "a")).kbenoit commentedon Jan 2, 2017
Or: We make the space the "higher precedence operator" where the tokenization occurs first, so that
keywords = c("is it|in a")matches "is it a" and "is in a". That was the original idea.koheiw commentedon Jan 2, 2017
After my own comment, I added a conversion function that allow
keywords = c("is it"), ("in a"). It is easy to split a string by|but we should not try to parse regular expression patterns, because it is too complex to do. Use of regular expression in this way would also mislead the users.kbenoit commentedon Jan 4, 2017
Solved in 2075c21