Bulk importing of CSV files

[clibassi]

Ported from a discussion on Slack - was hoping there might be a way to combine multiple CSV files at once that will ultimately be appended to each other. Current options are multiple CSV.read() statements or:

df = CSV.read(firstfile)
for file in rest
append!(df, CSV.File(file)
end

[quinnj]

Yeah, my thought here is we could have a CSV.File(::Vector) method that took a vector of filenames (or IO objects, or byte vectors), loop through each to do parsing, and append the results in a single CSV.File w/ ChainedVectors for each column, with chains being each file.

[kmundnic]

Is CSV.File(::Vector) necessary? I use dot notation for this:

df = vcat(CSV.read.(files, DataFrame)...)

[rofinn]

That solution requires a dependency on DataFrames.jl. I think if CSV.jl handled this internally it could be a lot more efficient than splatting the results into a vcat call. That operation would be very slow if you had hundreds of csv files. Alternatively, if Tables.Partitioner was also a valid table then you could just do table = Tables.Partitioner(CSV.File, myfiles) and if you want a DataFrame later on then DataFrame(table) could just work?

[quinnj]

DataFrames.jl has a specialized method on vcat that makes it pretty efficient. We already have TableOperations.joinpartitions that takes a valid Tables.partitions input and produces a "table", so you could do df = Tables.Partitioner(CSV.File, myfiles) |> TableOperations.joinpartitions |> DataFrame. joinpartitions operates lazily.

[rofinn]

Okay, I'll open a PR to TableOperations.jl to add that to the README.md

[quinnj]

Ok, PR is up to allow passing a vector of inputs to CSV.File or CSV.read and have them vertically, lazily concatenated by CSV.jl directly. I outline the reasoning for having this builtin to CSV.jl instead of using generic functionality in the PR. #873

[quinnj]

Implemented in #873