Skipping missing values more easily

[nalimilan]

It seems that dealing with missing values is one of the most painful issues we have, which goes against the very powerful and convenient DataFrames API. Having to write things like filter(:col => x -> coalesce(x > 1, false), df) or combine(gd, :col => (x -> sum(skipmissing(x))) isn't ideal. One proposal to alleviate this is #2258: add a skipmissing argument to functions like filter, select, transform and combine to unify the way one can skip missing values, instead of having to use different syntaxes which are hard to grasp for newcomers and make the code more complex to read.

That would be one step towards being more user-friendly, but one would still have to repeat skipmissing=true all the time when dealing with missing values. I figured two solutions could be considered to improve this:

• Have DataFramesMeta simplify the handling of missing values. This could be via a dedicated e.g. @linqskipmissing macro or a statement like skipmissing within a @linq block that would automatically pass skipmissing=true to all subsequent operations in a chain or block. This wouldn't really help with operations outside such blocks though.
• Have a field in DataFrame objects that would store the default value to use for the skipmissing argument. By default it would be false, so that you get the current behavior, which ensures safety via propagation of missing values. But when you know you are working with a data set with missing values, you would be able to call skipmissing!(df, true) once and then avoid repeating it.

Somewhat similar discussions have happened a long time ago (but at the array rather than the data frame level) at JuliaStats/DataArrays.jl#39. I think it's fair to say that we know have enough experience now to make a decision. One argument against implementing this at the DataFrame level is that it will have no effect on operations applied directly to column vectors, like sum(df.col). But that's better than nothing.

Cc: @bkamins, @matthieugomez, @pdeffebach, @mkborregaard

[pdeffebach]

• Have DataFramesMeta simplify the handling of missing values. This could be via a dedicated e.g. @linqskipmissing macro or a statement like skipmissing within a @linq block that would automatically pass skipmissing=true to all subsequent operations in a chain or block. This wouldn't really help with operations outside such blocks though.

This is a good idea, however I also like skipmissing = true at the level of a transform call or even at the level of argument because it's explicit.

Perhaps DataFramesMeta could provide a block-level skipmissing option as well as a macro like

@transform(df, @sm y = :x1 .+ mean(:x2))

Where @sm macro, does the necessary transformation described in #2258

function wrapper(fun, x, y)
    sx, sy = Missings.skipmissings(x, y) # need to be on Missings master
    sub_out = fun(sx, sy)
    full_out = Vector{Union{eltype(sub_out), Missing}}(missing, length(x))
    full_out[eachindex(sx)] .= sub_out # eachindex(sx) returns indices of complete cases

    return full_out
end

• Have a field in DataFrame objects that would store the default value to use for the skipmissing argument. By default it would be false, so that you get the current behavior, which ensures safety via propagation of missing values. But when you know you are working with a data set with missing values, you would be able to call skipmissing!(df, true) once and then avoid repeating it.

I'm not a fan of this idea since it's behavior depending on a global state that could be set a long ways from where the transform call is. It seems like it could cause debugging to be a pain.

[matthieugomez]

It's great you're thinking about how to make working with missing values easier! I 100% agree.

A macro may be good.

I'm not sure I'm convinced by an option at the level of the dataframe. It sounds complicated to keep track of it. For instance, won't people be confused that stuff like df = merge(df, df_using) does not retain the option?

A third possibility would be to allow users to change the default option for transform, etc, say by writing SKIPMISSING = true at the start of a script.

[c42f]

To throw in a much more crazy idea: could we use contextual dispatch to override the behavior of all reductions and comparisons within a whole expression so that they are "missing-permissive"?

I've long been frustrated with the difficulty of working with missing given that it infects all downstream operations and wished it worked differently in Base. (I acknowledge my frustration could be misguided — perhaps it's saved me from some horrible bugs without knowing it :-) )

[nalimilan]

Perhaps DataFramesMeta could provide a block-level skipmissing option as well as a macro like

@transform(df, @sm y = :x1 .+ mean(:x2))

Where @sm macro, does the necessary transformation described in #2258

@pdeffebach Yes that was more or less what I had in mind. Though if you have to repeat this for each call it's not much better than passing skipmissing=true. Being able to apply it to a chain of operations would already be more useful.

I'm not a fan of this idea since it's behavior depending on a global state that could be set a long ways from where the transform call is. It seems like it could cause debugging to be a pain.

@pdeffebach Yes. OTOH it's not so different from e.g. creating a column: if you get an error because it doesn't exist or it's incorrect you have to find where it's been defined, which can be quite far from where the error happens.

I'm not sure I'm convinced by an option at the level of the dataframe. It sounds complicated to keep track of it. For instance, won't people be confused that stuff like df = merge(df, df_using) does not retain the option?

A third possibility would be to allow users to change the default option for transform, etc, say by writing SKIPMISSING = true at the start of a script.

@matthieugomez Yes, losing the option after transformations could be annoying. Though it could be propagated across some operations which always preserve missing values: in these cases there's little point in forcing you to repeat that you know there are missing values. Where safety checks matter is when you could believe you got rid of missing values and for some reason it's not the case. But I admit that this option would decrease safety (even if not propagated automatically) as you could pass a data frame to a function which isn't prepared to handle missing values and it would silently skip them.

A global setting would only aggravate these issues IMHO since it would affect completely unrelated operations, possibly in packages, some of which may rely on the implicit skipmissing=false.

To throw in a much more crazy idea: could we use contextual dispatch to override the behavior of all reductions and comparisons within a whole expression so that they are "missing-permissive"?

@c42f My suggestion about DataFramesMeta is a kind of limited way to change the behavior of a whole block. Using Cassette.jl (which I guess is what you mean by "contextual dispatch"?) would indeed be a more general approach and it has been mentioned several times. Actually I just tried that and it turns out to be very simple to make it propagate missing values:

using Cassette, Missings

Cassette.@context PassMissingCtx

Cassette.overdub(ctx::PassMissingCtx, f, args...) = passmissing(f)(args...)

# do not lift special functions which already handle missing
for f in (:ismissing, :(==), :isequal, :(===), :&, :|, :⊻)
    @eval begin
        Cassette.overdub(ctx::PassMissingCtx, ::typeof($f), args...) = $f(args...)
    end
end

f(x) = x > 0 ? log(x) : -Inf

julia> Cassette.@overdub PassMissingCtx() f(missing)
missing

julia> Cassette.@overdub PassMissingCtx() f(1)
0.0

julia> Cassette.@overdub PassMissingCtx() ismissing(missing)
true

julia> Cassette.@overdub PassMissingCtx() missing | true
true

Skipping missing values in reductions will be a little harder, but it's doable if we only want to handle a known list of functions. For example this quick hack works:

Cassette.overdub(ctx::PassMissingCtx, ::typeof(sum), x) = sum(skipmissing(x))

julia> x = [1, missing];

julia> Cassette.@overdub PassMissingCtx() sum(x)
1

Maybe this kind of thing could be made simpler to use by providing a macro like @passmissing as a shorthand for Cassette.@overdub PassMissingCtx(). But it's important to measure all the implications of this approach: since it will affect all function calls deep in the code that you didn't write yourself, it will have lots of unintended side effects. For example, Cassette.@overdub PassMissingCtx() [missing] gives missing, which will break lots of package code. A safer approach would be to only apply passmissing to a whitelist of functions for which it makes sense (scalar functions, mainly), like DataValues does -- with the drawback that the list is kind of arbitrary.

In the end, maybe Cassette is too powerful for what we actually need in the context of DataFrames. In practice with select/transform/combine it makes sense to only apply passmissing to the top-level functions, rather than recursively. And for reductions, passing views of complete rows as proposed at #2258 should be enough, when combined with a convenient DataFramesMeta syntax.

[c42f]

My suggestion about DataFramesMeta is a kind of limited way to change the behavior of a whole block. Using Cassette.jl (which I guess is what you mean by "contextual dispatch"?) would indeed be a more general approach and it has been mentioned several times. Actually I just tried that and it turns out to be very simple to make it propagate missing values:

Very cool. IIUC people have started to use other libraries like IRTools which plug into the compiler in a similar way to Cassette but are not Cassette itself, but yes, that's roughly what I had in mind. One downside is that it's a pretty heavy weight tool to deploy for something like missing.

since it will affect all function calls deep in the code that you didn't write yourself, it will have lots of unintended side effects

I think this is the bigger problem; it's not clear how deep to recurse and it could definitely have unintended consequences. It's a very similar problem with floating point rounding modes, where having the rounding mode as dynamic state really doesn't work well as it infects other calculations which weren't programmed to deal with a different rounding mode.

So I agree it does seem much safer and more sensible to have a macro which lowers only the syntax within the immediate expression to be more permissive with missing. Actually I wonder whether you could do something more like broadcast lowering where all function calls within the expression are lifted in a certain way such that dispatch can be customized, so, eg, @linq f(x,y,z) becomes linq_missing(f, x, y, z).

In terms of your examples,

@linq filter(:col =>(x -> x > 1), df)
# means
linq_missing(filter, :col => (x -> linq_missing(>, x, 1)), df)

@linq combine(gd, :col => (x -> sum(x))
# means
linq_missing(combine, gd, :col => (x -> linq_missing(sum, x)))

Something like this has very regular lowering rules and gives a measure of extensibility for user defined functions.

[matthieugomez]

One alternative is that filter/transform/combine could always skip missing (i.e. kwarg skipmissing = true is the default).

[bkamins]

Can you please help me understanding this proposal in detail? What should be the outcome of the following operations if we added skipmissing option (now I am showing what we get now and would like to understand what we should get if we add skipmissing=true):

julia> df = DataFrame(x=[1,missing,2])
3×1 DataFrame
│ Row │ x       │
│     │ Int64?  │
├─────┼─────────┤
│ 1   │ 1       │
│ 2   │ missing │
│ 3   │ 2       │

julia> select(df, :x)
3×1 DataFrame
│ Row │ x       │
│     │ Int64?  │
├─────┼─────────┤
│ 1   │ 1       │
│ 2   │ missing │
│ 3   │ 2       │

julia> select(df, :x => identity)
3×1 DataFrame
│ Row │ x_identity │
│     │ Int64?     │
├─────┼────────────┤
│ 1   │ 1          │
│ 2   │ missing    │
│ 3   │ 2          │

julia> select(df, :x => ByRow(identity))
3×1 DataFrame
│ Row │ x_identity │
│     │ Int64?     │
├─────┼────────────┤
│ 1   │ 1          │
│ 2   │ missing    │
│ 3   │ 2          │

julia> combine(df, :x)
3×1 DataFrame
│ Row │ x       │
│     │ Int64?  │
├─────┼─────────┤
│ 1   │ 1       │
│ 2   │ missing │
│ 3   │ 2       │

julia> combine(df, :x => identity)
3×1 DataFrame
│ Row │ x_identity │
│     │ Int64?     │
├─────┼────────────┤
│ 1   │ 1          │
│ 2   │ missing    │
│ 3   │ 2          │

julia> combine(df, :x => ByRow(identity))
3×1 DataFrame
│ Row │ x_identity │
│     │ Int64?     │
├─────┼────────────┤
│ 1   │ 1          │
│ 2   │ missing    │
│ 3   │ 2          │

[nalimilan]

AFAICT this question regards #2258, better keep matters separate as discussions are already tricky enough.

[matthieugomez]

With `select`, `skipmissing` would apply the functions on row for which none of the argument is missing, return the output on these rows, and missing otherwise (to maintain the same number of rows as the original dataframe). So `skipmissing = true` would not change anything in your example. Similarly for `transform`. With `combine`, `skipmissing` would apply the functions on rows for which none of the argument is missing. ```julia julia> combine(df, :x) 3×1 DataFrame │ Row │ x │ │ │ Int64? │ ├─────┼─────────┤ │ 1 │ 1 │ │ 2 │ 2 │ julia> combine(df, :x => identity) 3×1 DataFrame │ Row │ x_identity │ │ │ Int64? │ ├─────┼────────────┤ │ 1 │ 1 │ │ 2 │ 2 │ julia> combine(df, :x => ByRow(identity)) 3×1 DataFrame │ Row │ x_identity │ │ │ Int64? │ ├─────┼────────────┤ │ 1 │ 1 │ │ 2 │ 2 │ ``` That being said, I am not sure why `combine(df, :x)` or `combine(df, :x => ByRow(.))` are accepted and it sounds that maybe it would be clearer to disallow them?

[bkamins]

Ah - sorry - I thought this issue "overriden" that one. So actually this issue should be on hold till we resolve #2258, because it only asks to make #2258 simpler - right?