Pass ... to base::print

[koheiw]

Pass print.tokens(...) to base::print() to hide quotes around tokens. I think it is easier to read without quotes.

require(quanteda)
#> Loading required package: quanteda
#> Package version: 4.0.2
#> Unicode version: 15.1
#> ICU version: 74.1
#> Parallel computing: 8 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
toks <- tokens(data_corpus_inaugural[1:3])
print(toks)
#> Tokens consisting of 3 documents and 4 docvars.
#> 1789-Washington :
#>  [1] "Fellow-Citizens" "of"              "the"             "Senate"         
#>  [5] "and"             "of"              "the"             "House"          
#>  [9] "of"              "Representatives" ":"               "Among"          
#> [ ... and 1,525 more ]
#> 
#> 1793-Washington :
#>  [1] "Fellow"   "citizens" ","        "I"        "am"       "again"   
#>  [7] "called"   "upon"     "by"       "the"      "voice"    "of"      
#> [ ... and 135 more ]
#> 
#> 1797-Adams :
#>  [1] "When"      "it"        "was"       "first"     "perceived" ","        
#>  [7] "in"        "early"     "times"     ","         "that"      "no"       
#> [ ... and 2,565 more ]
print(toks, quote = FALSE)
#> Tokens consisting of 3 documents and 4 docvars.
#> 1789-Washington :
#>  [1] Fellow-Citizens of              the             Senate         
#>  [5] and             of              the             House          
#>  [9] of              Representatives :               Among          
#> [ ... and 1,525 more ]
#> 
#> 1793-Washington :
#>  [1] Fellow   citizens ,        I        am       again    called   upon    
#>  [9] by       the      voice    of      
#> [ ... and 135 more ]
#> 
#> 1797-Adams :
#>  [1] When      it        was       first     perceived ,         in       
#>  [8] early     times     ,         that      no       
#> [ ... and 2,565 more ]

Ideally, there is a option to print (or coerce) like this. But I don't know what is the best way.

lapply(toks, function(x) paste0(head(x, 12), collapse = " "))
#> $`1789-Washington`
#> [1] "Fellow-Citizens of the Senate and of the House of Representatives : Among"
#> 
#> $`1793-Washington`
#> [1] "Fellow citizens , I am again called upon by the voice of"
#> 
#> $`1797-Adams`
#> [1] "When it was first perceived , in early times , that no"

[kbenoit]

Ah... that's a somewhat radical change. Makes it especially hard to distinguish tokens that contain spaces. What if we added this as an option, with the default to keep the quotes?

[koheiw]

I agree that tokens should be quoted by default but it is nice to think how to make it easier to read. For example,

#>  [1] "Fellow-Citizens" "of" "the" "Senate"         
#>  [5] "and" "of" "the" "House"          
#>  [9] "of" "Representatives" ":" "Among" 

Not urgent, though...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.17%. Comparing base (15eb2df) to head (b16387b).
Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2381      +/-   ##
==========================================
- Coverage   96.18%   96.17%   -0.01%     
==========================================
  Files          94       94              
  Lines        5892     5890       -2     
==========================================
- Hits         5667     5665       -2     
  Misses        225      225              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kbenoit]

This is pretty innocuous and only turns on if people want it to. We could add a global option print_tokens_quote and set it to TRUE for the default behaviour. Or, just leave it as you have here.