Exporting DataFrame to CSV also dumps array datatype

[captchanjack]

I'm noticing a bug (possibly just a feature that can't be turned off?) where exporting a DataFrame to CSV also dumps the datatype of arrays:

Is there a way to not dump the datatypes? This makes it hard to parse and read the csv again.

[quinnj]

Currently, CSV.jl isn't built to handle writing/reading arrays/lists of values at all. I can imagine one way to do this would be a custom type, something like:

struct ListScalar{T, delim}
    values::Vector
end

Base.zero(::Type{ListScalar{T, delim}}) where {T, delim} = ListScalar{T, delim}(T[]) 
Base.string(x::ListScalar{T, delim}) where {T, delim} = join(x.values, delim)
Base.parse(::Type{ListScalar{T, delim}}, str) where {T, delim} = ListScalar{T, delim}(map(x -> parse(T, x), split(str, delim)))

So then you could do:

julia> struct ListScalar{T, delim}
           values::Vector
       end

julia> Base.zero(::Type{ListScalar{T, delim}}) where {T, delim} = ListScalar{T, delim}(T[])

julia> Base.string(x::ListScalar{T, delim}) where {T, delim} = join(x.values, delim)

julia> Base.parse(::Type{ListScalar{T, delim}}, str) where {T, delim} = ListScalar{T, delim}(map(x -> parse(T, x), split(str, delim)))

julia> using CSV

julia> x = (x=[ListScalar{Int, ';'}([1,2,3,4,5])],)
(x = ListScalar{Int64, ';'}[ListScalar{Int64, ';'}([1, 2, 3, 4, 5])],)

julia> CSV.write("listscalars.csv", x)
"listscalars.csv"

julia> f = CSV.read("listscalars.csv"; delim=',', types=[ListScalar{Int, ';'}])
1×1 DataFrame
│ Row │ x                                       │
│     │ ListScalar{Int64, ';'}                  │
├─────┼─────────────────────────────────────────┤
│ 1   │ ListScalar{Int64, ';'}([1, 2, 3, 4, 5]) │

[captchanjack]

Nice solution! though this seems like quite a work around wrapping all arrays in the custom type, regardless thanks for putting this response together, I'll use this for now :)!

[nalimilan]

This might be a bug in Julia actually: print(Signed[1, 2]) gives Signed[1, 2], but ?print says "The representation used by print includes minimal formatting and tries to avoid Julia-specific details." Yet Signed is clearly a Julia-specific detail. Probably worth filing an issue against Julia to discuss this.

[hs-ye]

Running into this problem as well, and this significantly reduces the utility of CSV+Dataframes because of the non-standard csv writing of arrays.

@nalimilan are you saying that CSV uses the julia print function to write into file? That feels like asking for trouble if core Julia decides to change their implementation at any stage

[quinnj]

Wow, I'm actually really surprised to see people relying on an accidental feature of CSV.write in that it would just call string(x) and write out whatever is returned there to the file. It's curious because, as far as I can guess, CSV.read won't really read those values back in? Are you exporting these arrays of arrays to some other program to read? Post-processing the columns yourself after writing?

This is a case where using something like https://github.com/JuliaData/Arrow.jl will probably help your workflows immensely. Arrow has native support for arrays of arrays as columns and stores them efficiently.