Automated csv/tsv parsing

[sandytribal]

Even with a UI allowing the user to tell the cli whether the file is comma separated or space separated, tab separated, etc. some users are getting tripped up. The user who reported getting tripped up was a programmer himself. I fear that users who are less technical, and don't even know what the difference between csv and tsv is, will fail to get past this step.

Is there a way to simply detect what format the file is in and then parse it?

[KristijanArmeni]

DuckDB (a popular and mature database management tool) has a csv sniffer tool that does exactly this. From their blog:

CSV files offer the advantage of flexibility; they are structured as text files, allowing users to manipulate them with any text editor, and nearly any data system can read and execute queries on them. [...] However, this flexibility comes at a cost. Reading a CSV file is not a trivial task, as users need a significant amount of prior knowledge about the file. For instance, DuckDB's CSV reader offers more than 25 configuration options. These options include specifying the delimiter, quote and escape characters, determining the number of columns in the CSV file, and identifying whether a header is present while also defining column types. This can slow down an interactive data exploration process, and make analyzing new datasets a cumbersome and less enjoyable task.

DuckDB implements a multi-hypothesis CSV sniffer that automatically detects dialects, headers, date/time formats, column types, and identifies dirty rows to be skipped. Our ultimate goal is to automatically read anything resembling a CSV file, to never give up and never let you down!

I'd be happy to look into this once we get the #194 merged in.