Some tricky data for Arrow

[xiaodaigh]

I have this data in the JDF.jl tests. Arrow is currently failing to write it. We could incorporate something like this in the tests.

using JDF
using DataFrames
using Random: randstring
using WeakRefStrings

df = DataFrame([collect(1:100) for i = 1:3000])
df[!, :int_missing] =
    rand([rand(rand([UInt, Int, Float64, Float32, Bool])), missing], nrow(df))

df[!, :missing] .= missing
df[!, :strs] = [randstring(8) for i = 1:nrow(df)]
df[!, :stringarray] = StringVector([randstring(8) for i = 1:nrow(df)])

df[!, :strs_missing] = [rand([missing, randstring(8)]) for i = 1:nrow(df)]
df[!, :stringarray_missing] =
    StringVector([rand([missing, randstring(8)]) for i = 1:nrow(df)])
df[!, :symbol_missing] = [rand([missing, Symbol(randstring(8))]) for i = 1:nrow(df)]
df[!, :char] = getindex.(df[!, :strs], 1)
df[!, :char_missing] = allowmissing(df[!, :char])
df[rand(1:nrow(df), 10), :char_missing] .= missing

@time JDF.save("a.jdf", df) # works with JDF 


# but fails with Arrow
Arrow.write("a.arrow", df)

error msg

4,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Int64,Union{Missing, UInt64},Missing,String,String,Union{Missing, String},Union{Missing, String},Union{Missing, Symbol},Char,Union{Missing, Char}}}, ::DataFrame, ::Dict{Int64,Tuple{Int64,Type,Any}}) at C:\Users\RTX2080\.julia\packages\Arrow\FjbLX\src\write.jl:258 [9] macro expansion at C:\Users\RTX2080\.julia\packages\Arrow\FjbLX\src\write.jl:71 [inlined] [10] macro expansion at .\task.jl:332 [inlined] [11] write(::IOStream, ::DataFrame, ::Bool, ::Bool) at C:\Users\RTX2080\.julia\packages\Arrow\FjbLX\src\write.jl:58 [12] #29 at C:\Users\RTX2080\.julia\packages\Arrow\FjbLX\src\write.jl:21 [inlined] [13] open(::Arrow.var"#29#30"{Bool,DataFrame}, ::String, ::Vararg{String,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at .\io.jl:325 [14] open at .\io.jl:323 [inlined] [15] #write#28 at C:\Users\RTX2080\.julia\packages\Arrow\FjbLX\src\write.jl:20 [inlined] [16] write(::String, ::DataFrame) at C:\Users\RTX2080\.julia\packages\Arrow\FjbLX\src\write.jl:20 [17] top-level scope at REPL[44]:1 [18] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1088

[quinnj]

It's helpful to post backtraces as that can often quickly show where the error is

[xiaodaigh]

It prints out alot of stuff. It's like 11pm. I will try to get a MWE together.

[quinnj]

Alright, so the error here is because it doesn't know how to map Symbol or Char columns to arrow format. I guess in both cases, we should just map them to strings.

[quinnj]

I guess the biggest question is if we're ok not having round-tripping in these cases; i.e. we could allow writing columns of Symbol/Char, but they'll be read back in as strings. @bkamins @nalimilan , any thoughts/preferences here?

[nalimilan]

Not sure. Maybe see what Feather.jl does? Also of interest is what RCall should do with CategoricalArray{T} when !(T<:AbstractString), as R only supports string factors (JuliaInterop/RCall.jl#331).

[bkamins]

I think the priority is to allow saving things with as little problems as possible. Symbol and Char are normal in Julia.

Given your comment I understand that we have a choice:

1. either error on them
2. or convert to String when saving

I would prefer the option 2. maybe with a @warn about conversion

[quinnj]

I mean, Feather.jl will just be a wrapper around Arrow.jl pretty soon, so they'll do whatever we do. I also think I like the idea of option 2 w/ a warning. I think people should understand that Arrow.jl isn't meant to be a serialization package, even though it can handle a wide range of types of objects. But even still, I think doing some automatic conversions will be convenient.

[bkamins]

Agreed

[xiaodaigh]

There is a way to store arbitrary metadata right? We should remember them as Julia:Symbol and Julia:Char types. So in Python and r, they get read as String but in Julia we can use the metadata to restore them as Symbol and Char?

[quinnj]

Ah, I forgot about that; yes, that's a pretty good idea. I'll have to see how we can work the plumbing in for that.

[quinnj]

Ok, took a bit of work, but I think I've got it all working for Symbol/Char. #11