Skip to content

kwic regex match not working fully #307

Closed

Description

@kbenoit

(No doubt this is also related to #306.)

require(quanteda)
## Loading required package: quanteda
## quanteda version 0.9.8.12

mycorpus <- corpus(c("This is a test",
                     "This is it.",
                     "What is in a train?",
                     "Is it a question?",
                     "Sometimes you don't know if this is it.",
                     "Is it a bird or a plane or is it a train?"))
                   
# CORRECT FAIL: because this is parsed as three tokens, with no match
# for the middle token, "is|sometimes"
kwic(mycorpus, keywords = c("this is|sometimes you"), valuetype = "regex")
## [1] NA


# INCORRECT FAIL: should also match text2
kwic(mycorpus, keywords = c("is it|in a"), valuetype = "regex")
##  [text4, 1:3]                    [ Is it a ] question?         
##  [text6, 1:3]                    [ Is it a ] bird or a plane or
## [text6, 9:11] bird or a plane or [ is it a ] train? 

# INCORRECT FAIL: should also match text3
kwic(mycorpus, keywords = c("is it|in a"), valuetype = "regex")
##  [text4, 1:3]                    [ Is it a ] question?         
##  [text6, 1:3]                    [ Is it a ] bird or a plane or
## [text6, 9:11] bird or a plane or [ is it a ] train? 

# INCORRECT FAIL: should also match text3
kwic(mycorpus, keywords = c("is i? a"), valuetype = "glob")
##  [text4, 1:3]                    [ Is it a ] question?         
##  [text6, 1:3]                    [ Is it a ] bird or a plane or
## [text6, 9:11] bird or a plane or [ is it a ] train? 

# INCORRECT FAIL: Individual matches should work in a vector too
# --- this is ok:
kwic(mycorpus, keywords = c("is it"), valuetype = "regex")
##                             contextPre keyword         contextPost
##  [text2, 2:3]                   This [   is it ]                  
##  [text3, 2:3]                   What [   is it ]                  
##  [text4, 1:2]                        [   Is it ] a question?      
##  [text5, 7:8] you don't know if this [   is it ]                  
##  [text6, 1:2]                        [   Is it ] a bird or a plane
## [text6, 9:10]     bird or a plane or [   is it ] a train?  
# --- this is ok:
kwic(mycorpus, keywords = c("or a"), valuetype = "regex")
##                  contextPre keyword        contextPost
## [text6, 5:6] Is it a bird [    or a ] plane or is it a
# -- NOT OK:
kwic(mycorpus, keywords = c("is it", "or a"), valuetype = "regex")
## [1] NA

Thanks to @jjphorn for bringing this to my attention.

Activity

  1. self-assigned this
    on Nov 15, 2016
  2. modified the milestone: v0.9.9 release on Dec 30, 2016
  3. added a commit that references this issue on Jan 2, 2017
  4. koheiw commented on Jan 2, 2017

    @koheiw
    Collaborator

    I do not think we should allow use of | operator. keywords = c("is it|in a") should be keywords = c("is it"), ("in a"), or more strictly keywords = list(c("is", "it"), c("in", "a")).

  5. kbenoit commented on Jan 2, 2017

    @kbenoit
    CollaboratorAuthor

    Or: We make the space the "higher precedence operator" where the tokenization occurs first, so that keywords = c("is it|in a") matches "is it a" and "is in a". That was the original idea.

  6. koheiw commented on Jan 2, 2017

    @koheiw
    Collaborator

    After my own comment, I added a conversion function that allow keywords = c("is it"), ("in a"). It is easy to split a string by | but we should not try to parse regular expression patterns, because it is too complex to do. Use of regular expression in this way would also mislead the users.

  7. kbenoit commented on Jan 4, 2017

    @kbenoit
    CollaboratorAuthor

    Solved in 2075c21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions