-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How are characters like tab and new line handled? #105
Comments
By default, TAB and newline characters are replaced a space. A different replacement string can be specified using the These details can be found in the csv2tsv reference, or by using the |
But doesn't this change the original data? Tabs and newlines might special means in the original CSV files. Generally speaking, are there field delimiters and line delimiters that are safe to use in most situations? That is people seldom use them in text and no escape is required for them. |
TSV file format uses TABs and newlines as delimiters precisely because these characters are not part of the data in the vast majority of large data sets. It's certainly true of most data mining scenarios. The reason Of course, TAB and newline may carry meaning in some data sets. Most software tools support alternate field delimiters, so something other than TAB can be chosen. Very few tools support alternate record (line) delimiters. If newlines need to be preserved in the data set being used, then it's better to use a file format that supports full character escaping. Using TSV is a tradeoff. The lack of escapes results in much simpler and less error prone processing, and can run faster as well. However, it is not the best format for every use case. To your more general question: ASCII does define control characters intended for these purposes, CTRL-~ and CTRL-_ (decimal 30 and 31, Hex 1E and 1F) for record separator and field separator. However, these never got much support. However, you use one of these in place of TAB for field delimiter. There is related discussion in the Comparing TSV and CSV formats page in this repository. The wikipedia article on Tab-separated values has more info on TSV file format. Update: In Release 2.1.0 |
Been 13 days, assuming the answer is sufficient and closing. Please add addition comments if further discussion would benefit. |
When converting a CSV file to TSV? I assume there are either default alternative characters or users have to specify alternative filed/line delimiters?
The text was updated successfully, but these errors were encountered: