Fix text delimiter

[lhoestq]

I changed the delimiter in the text dataset script.
It should fix the pyarrow.lib.ArrowInvalid: CSV parse error from #622

I changed the delimiter to an unused ascii character that is not present in text files : \b

[thomwolf]

Youpi 🎶

[abhi1nandy2]

Got this error on using the delimiter \b. Resolved this error by reverting back to \r. Don't know why though!

pyarrow.lib.ArrowInvalid: CSV parse error: Expected 1 columns, got 4

[lhoestq]

Which OS are you using ?@abhi1nandy2

[abhi1nandy2]

Which OS are you using ?

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[lhoestq]

Do you mind sharing the data you used (or part of it), so I can try to reproduce ?
Or at least some info about the text file you're using ? (size, n of lines, encoding)

[abhi1nandy2]

Lot of data, difficult to share. There are 46 shards, each having about 256000 lines. using file command gives this - ASCII text, with very long lines.

[lhoestq]

Ok I see, no problem :)
I'll see what I can do

Could you just test with one single dummy text file with a few lines to see if you're having the issue ?
Also which version of datasets do you have ?