unstack error message for missing values

[jariji]

julia> @chain begin
           DataFrame(x=rand(1:100, 10), y=rand(['a','b', missing], 10), z=rand(10))
           unstack(:x, :y, :z; )
       end
ERROR: ArgumentError: Missing value in variable :y. Pass `allowmissing=true` to skip missings.

"Pass allowmissing=true to skip missings" Is that an accurate description of what that kwarg does? The docs say "if true then a column referring to missing value is created" which seems like a different thing.

[bkamins]

Yes, it should be fixed. But while fixing it let us decide how this case should be handled:

julia> unstack(DataFrame(x=rand(1:100, 10), y=rand(["missing", missing], 10), z=rand(10)), :x, :y, :z, allowmissing=true)
ERROR: ArgumentError: Duplicate variable names: :missing. Pass makeunique=true to make them unique using a suffix automatically.

Maybe we should allow passing column name where missing should be mapped?

[bkamins]

Also we have the following issue:

julia> unstack(DataFrame(x=rand(1:100, 10), y=rand(['a', "a"], 10), z=rand(10)), :x, :y, :z, allowmissing=true)
ERROR: ArgumentError: Duplicate variable names: :a. Pass makeunique=true to make them unique using a suffix automatically.

I am not sure if we want to error in this case, or allow for some other behavior also.

[jariji]

Why does missing need special handling? Why not just treat it like a normal value/column?

[bkamins]

It does not. I am just saying that I noticed that we need to decide what to do if we have two distinct values that map to the same column name, e.g. "missing" and missing, or 'a' and "a". Maybe we need a separate kwarg.

[jariji]

I mean what is the point of allowmissing=false at all - why not just do it?

[bkamins]

Because, conceptually, missing can be any value, so we do not know to what value we should map it to.
If allowmissing=true we say that missing should be treated as a valid value (and not as a sentinel for a value that we do not know).

[jariji]

Can you distinguish that reasoning from sort([1,2,missing]) which doesn't require an allowmissing=true kwarg? missing can appear in any position, so first(sort(xs)) could give the wrong answer but Julia says that's okay.

[bkamins]

sort design is not missing aware, while DataFrames.jl design is.