When vcat dataframes, ordering of categorical variables is lost

[daisy12321]

vcat on the categorical array preserves the ordering, but on the dataframe it is lost. Example below:

using DataFrames
df = DataFrame()
df.cat = CategoricalArray(rand(["a", "b", "c"], 100), ordered=true)

outputs:

julia> [df; df].cat[1]
CategoricalString{UInt32} "b"

julia> [df.cat; df.cat][1]
CategoricalString{UInt32} "b" (2/3)

[bkamins]

Indeed it would be better to ensure that we properly preserve this. Probably @nalimilan has most experience to fix this, as this is a tricky case.

[nalimilan]

Good catch. The problem is that we don't call vcat on the vectors, but allocate the new column upfront and fill it using copyto!. Unfortunately, we can't use vcat since it doesn't support promotion of array types (JuliaLang/julia#20815), and because we need to insert missing values if the column doesn't exist in one of the data frames.

But maybe we can change copyto! to fix that. Currently we check whether the destination is ordered, and since a newly allocated array isn't, we never mark it as ordered even if the source is. But we could change the condition so that we also mark it as ordered if it has no levels and the source is ordered. In that case, we would also change setindex! to do the same. That would add a special case, but it might not be a problem since an array with no levels is clearly quite special anyway.

[nalimilan]

I have filed JuliaData/CategoricalArrays.jl#223 to implement this, please have a look. I've realized that we already had a similar special case in CategoricalArrays' vcat method: arrays with empty levels are ignored when checking whether all input arrays are ordered.

Another solution would be to mark all CategoricalArray objects as ordered when they are created uninitialized. That would be OK if we automatically marked them as unordered when adding levels to them (except when assigning from an ordered categorical value). But that approach doesn't seem to be preferable to JuliaData/CategoricalArrays.jl#223, as marking arrays as ordered by default could be slightly confusing.