importing tab-separated data

[ekg]

@arq5x and I were trying to import a BED file into dat:

dat init
...
curl http://quinlanlab.cs.virginia.edu/cshl2013/cpg.bed > cpg.bed
cat cpg.bed | dat import --csv --separator '\t'

This yields:

buffer.js:188                                                                                       
    throw new TypeError('First argument needs to be a number, ' +                               
          ^                                                                                     
TypeError: First argument needs to be a number, array or string.                                    
at new Buffer (buffer.js:188:15)                                                                
at new Parser (/usr/local/lib/node_modules/dat/node_modules/csv-parser/index.js:17:52)          
at module.exports (/usr/local/lib/node_modules/dat/node_modules/csv-parser/index.js:163:10)     
at parseCSV (/usr/local/lib/node_modules/dat/lib/write-stream.js:121:7)                         
at parseStream (/usr/local/lib/node_modules/dat/lib/write-stream.js:98:44)                      
at writeStream (/usr/local/lib/node_modules/dat/lib/write-stream.js:17:16)                      
at Object.dat.createWriteStream (/usr/local/lib/node_modules/dat/lib/commands.js:737:10)        
at module.exports (/usr/local/lib/node_modules/dat/bin/import.js:27:20)                         
at execCommand (/usr/local/lib/node_modules/dat/cli.js:66:22)                                   
at Server.<anonymous> (/usr/local/lib/node_modules/dat/lib/commands.js:230:11)                  

As a workaround, we convinced dat that the tab-separated data is csv:

cat ../cpg.bed | tr '\t' ',' | dat import --csv

Which made it happy.

[max-mapper]

two issues here, one is that we need to handle streaming import errors more gracefully (me and @finnp were talking about this yesterday)

second is that we dont handle \t but instead expect a tab literal, so to do that in bash you do $'\t' (reference)

this works:

$ dat import ../simple.tsv --separator=$'\t'

but we should probably add a special case for handling \t, or maybe theres a more general solution?

[arq5x]

Looks like we used --separator incorrectly. We did --separator $'\t' instead of (note the equal sign) --separator=$'\t'

[thadguidry]

@maxogden Does the --separator= handle multibyte sequences for the "field separator" ? If not, then an issue should be created.

Also, does dat support multi-byte sequences for the "record separator" ? If not then an issue should be created.

Example of multi-byte sequences for field and record separators: https://www.apple.com/itunes/affiliates/resources/documentation/itunes-enterprise-partner-feed.html#fileformat

[max-mapper]

@thadguidry this is our CSV test suite: https://github.com/maxogden/csv-spectrum if you wanna make a PR adding the apple example it would be very appreciated!

[joyrexus]

@maxogden wrote:

this works: $ dat import ../simple.tsv --separator=$'\t'

Minor point: I think you still need to include the --csv switch.

FWIW, it's also possible to insert an actual tab character using CTRL-V (control + V keys) followed by TAB (tab key):

cat simple.tsv | dat import --csv --separator='     '

@maxogden would you be open to a simple PR for autodetecting .tsv file extensions and setting the separator accordingly?

[max-mapper]

@joyrexus yes that is a good idea, @finnp wrote the code you mentioned in case you have specific questions. Just make sure you add a test!

[finnp]

@joyrexus Right now tsv is automatically detected as a stream. So cat simple.tsv | dat import works. However dat import simple.tsv does not use the file extension, so it still uses input detection. So yeah it would be cool if you could make a PR for the tsv extension.

[thadguidry]

@maxogden Added the Apple EPF test cases (Multi-Separator & Multi-EOL Char Support) to pull in max-mapper/csv-spectrum#9

[joyrexus]

@finnp wrote:

Right now tsv is automatically detected as a stream. So cat simple.tsv | dat import works. However dat import simple.tsv does not use the file extension, so it still uses input detection.

The latter should work now with PR #229.

[finnp]

@joyrexus Yup, awesome 👍

[pkafei]

Yay for .tsv detection!

[ekg]

In PR #232 I add explicit support for tsv via a flag (--tsv).

I should have followed @finnp's progress with input detection, as it seems this comes a bit late. Explicit support does ensure that oddities like the following are handled correctly:

% echo 'a,is,cool\tb\tc\n1,2\t2\t3\n4\t5\t6\n7\t8\t9' | dat import
No import file specified, using STDIN as input
Parsing detected format csv with separator ","
Done

It also clarifies to users that tsv input is natively supported. (NB: #232 incorporates @joyrexus's #229.)

[max-mapper]

IMO our contributing.md should read like a checklist. So if we need to do a long form description of a concept it should be a ilnk to another .md file that has the long form explanation

Also the main things I wanna mention in the contributing.md are:

• follow style of the code that you are adding to (e.g. spaces not tables, no semicolons etc)
• we don't use a linter or have a js style guide
• a section on how to run the tests, and how to add new tests
• we prefer to remove code from the dat module when we can, rather than add in new code

[finnp]

@maxogden Wrong issue :)