What should we do with "--"

[koheiw]

There are many instances of repeated hyphens ("--" or "---") because people use them as dash ("—"). Repeated hyphenates are also preserved, but we don't really want them. Shall we substitute "[\\p{Pd}]{2,}" by "—" before preserve_special()?

[kbenoit]

Typography, it's a PITA. Three variants:

• "one-two" (we preserve this)
• "one -- two" (we preserves the "--" as a token)
• "one - two" (we preserve the "-" as a token)
• "one--two" is currently preserved entirely.

Of these, I think all are as should be, except the last. How about if preserve_special() just replaces a single hyphen and not more than one?

We should defo avoid using special hyphens, like the "em" dash . In fact we get rid of all these in

quanteda/R/corpus.R

Lines 154 to 155 in 0b98843

	# replace all hyphens with simple hyphen
	x <- stri_replace_all_regex(x, "\\p{Pd}", "-")

.
Those and curly quotes keep me awake at night!

[koheiw]

Actually, many instances are our own creation:

quanteda/R/corpus.R

Line 155 in 0b98843

x <- stri_replace_all_regex(x, "\\p{Pd}", "-")

We shouldn't do this anymore.

[kbenoit]

We replace it because we want to distill the essence of documents into text, not typography.

[koheiw]

How about cases like this?

> tokens("thanks—all look good!")
Tokens consisting of 1 document.
text1 :
[1] "thanks-all" "look"       "good"       "!" 

[kbenoit]

Absent a fancy language model, we have no real choice but to live with the fact that our pattern-based tokenizer treats that as an infix hyphen.

[koheiw]

How about doing this in corpus()?

x <- stri_replace_all_fixed(x, c("—", "–"), " - ", vectorlize_all = FALSE) 
x <- stri_replace_all_regex(x, "\\p{Pd}", "-") 

[kbenoit]

It's hard to differentiate the different hyphen characters in the code font, so I'm not 100% sure what those are here. The point though, is that we want (all are standard hyphens here)

"This--is text -- end."
to become
"This - is text - end."
?

[koheiw]

Yes, then we can remove the dash-like hyphen easily.

> tokens("This--is text -- end.")
Tokens consisting of 1 document.
text1 :
[1] "This--is" "text"     "--"       "end"      "."       

> tokens("This - is text - end.")
Tokens consisting of 1 document.
text1 :
[1] "This" "-"    "is"   "text" "-"    "end"  "."  

[kbenoit]

Then sounds good to me!