Do not update concatenator by tokens_compound()

[koheiw]

Currently, tokens_compound() updates the default concatenator but I don't think it should.

Currently

require(quanteda)
toks <- tokens(c(doc1 = "New York is over crowded"), concatenator = " ") %>% 
    tokens_compound(phrase("over crowded"), concatenator = "") %>% 
    tokens_compound(phrase("New York"))
print(toks)
#> Tokens consisting of 1 document.
#> doc1 :
#> [1] "NewYork"     "is"          "overcrowded"

Should be

require(quanteda)
toks <- tokens(c(doc1 = "New York is over crowded"), concatenator = " ") %>% 
    tokens_compound(phrase("over crowded"), concatenator = "") %>% 
    tokens_compound(phrase("New York")) # use the default concatenator
print(toks)
#> Tokens consisting of 1 document.
#> doc1 :
#> [1] "New York"     "is"          "overcrowded"

[kbenoit]

I encountered this before too. But given that the most common use case is to compound MWEs, it still seems to make sense to use " " as the default. Plus, this change could break a lot.

[koheiw]

When texts are mix of different languages (Japanese/Chinese and English), I need to use different concatenators. I think you agree that the proposed behavior is more correct, because there was no way to set the default concatenator in tokens() when tokens_compound() was created. If so, the question is how to make transition without upsetting too many users.

[koheiw]

How about warning when concat(x) is different from the value given to concatenator in tokens_compound()? For example, "concatenator does not match the tokens object's default concatenator; use tokens() to update the default value`.

It is risky to use different concatenator in the current tokens_compound() because the downstream pattern matching of phrases goes wrong.

require(quanteda)
#> Loading required package: quanteda
#> Package version: 4.1.0
#> Unicode version: 15.1
#> ICU version: 74.1
#> Parallel computing: 16 of 16 threads used.
#> See https://quanteda.io for tutorials and examples.
toks <- tokens(c(doc1 = "New York is over crowded"), concatenator = " ") 

toks <- tokens_compound(toks, phrase("New York"))
toks <- tokens_compound(toks, phrase("over crowded"), concatenator = "-")
    
print(toks)
#> Tokens consisting of 1 document.
#> doc1 :
#> [1] "New York"     "is"           "over-crowded"
concat(toks)
#> [1] "-"

tokens_lookup(toks, dictionary(list(NY = "New York")))
#> Tokens consisting of 1 document.
#> doc1 :
#> character(0)
tokens_lookup(toks, dictionary(list(NY = "New-York")))
#> Tokens consisting of 1 document.
#> doc1 :
#> character(0)
tokens_lookup(toks, dictionary(list(NY = "New-York"), separator = "-"))
#> Tokens consisting of 1 document.
#> doc1 :
#> character(0)

^{Created on 2024-12-16 with reprex v2.1.1}