Twitter shortened URLs are not removed

[plablo09]

The regular expression that removes URLs is expecting at least two characters in the domain name part. Since Twitter shortens URLs as https://t.co/something these URLs are not removed.

I fixed this in my own fork. The fix is quite simple, just change:

URLREGEX <- "https?:\\/\\/(www\\.)?[-a-zA-Z0-9@:%._\\+~#=]{2,256}\\.[a-z]{2,4}\\b([-a-zA-Z0-9@:%_\\+.~#?&//=]*)"

to

URLREGEX <- "https?:\\/\\/(www\\.)?[-a-zA-Z0-9@:%._\\+~#=]{1,256}\\.[a-z]{2,4}\\b([-a-zA-Z0-9@:%_\\+.~#?&//=]*)"

In both tokenize.R and tokens.R

Maybe a pull request?
Or this is just too simple and may be easier to fix directly on this repo

Regards

[kbenoit]

Thanks, I just fixed it!

[kbenoit]

Next time feel free to issue a short PR, since that gives you official credit on the GitHub system (if that is your thing).