[Rust] CSV Reader & Writer

[asfimport]

As an Arrow Rust user, I would like to be able to read and write CSV files, so that I can quickly ingest data into an Arrow format for futher use, and save outputs in CSV.

As there aren't yet many options for working with tabular/df structures in Rust (other than Andy's DataFusion), I'm struggling to motivate for this feature. However, I think building a csv parser into Rust would reduce effort for future libs (incl DataFusion).

Reporter: Neville Dipale / @nevi-me
Assignee: Andy Grove / @andygrove

PRs and other links:

• GitHub Pull Request #2992

_{Note: This issue was originally created as ARROW-3726. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Since @pitrou has just been building a fresh new parallel CSV reader for Cplusplus, its design could be used to help with creating a Rust-native one (or you could possibly wrap the C++ library in Rust, too)

[asfimport]

Chao Sun / @sunchao:
Looks interesting. I'll take this (probably with Rust-native impl).

[asfimport]

Andy Grove / @andygrove:
I have a CSV reader + writer already in DataFusion that I was planning on contributing. It needs to be updated to reflect some of the recent changes in the Rust implementation of Arrow.

@sunchao Are you OK with me taking this one, or do you still want to work on this?

[asfimport]

Andy Grove / @andygrove:
Also, worth stating that my reader + writer leverages the excellent (and high performance) Rust csv crate, so we don't have to reinvent the wheel there.

[asfimport]

Neville Dipale / @nevi-me:
Thanks @andygrove, does the reader support inferring data schema? I got schema inference working through sampling a csv and using the regex crate to match fields by some hierarchy. 

It worked relatively well on primitive types, but I struggled with timestamps because I was trying to read from csv to parquet. For timestamps we could allow the user to specify the format (yyyy-mm-dd-... etc), or default to an ISO format. What do you think?

[asfimport]

Andy Grove / @andygrove:
I have not implemented schema inference yet. I think we should create a separate ticket for this feature where we can figure out the requirements, like the timestamp issue you mentioned.

[asfimport]

Andy Grove / @andygrove:
Issue resolved by pull request 2992
#2992

[asfimport]

Neville Dipale / @nevi-me:
Hi @andygrove @pitrou, I just remembered that the csv library used in Arrow only supports UTF8. Does the cpp-based CSV reader have the same limitation, or will this be limited to Rust? It might be something worth documenting for Rust.