Consider allowing default in quantile and median

[bkamins]

In generic code the fact that the following operations error:

julia> median(Int[])
ERROR: ArgumentError: median of an empty array is undefined, Int64[]

julia> quantile(Int[], 0.5)
ERROR: ArgumentError: empty data vector

is inconvenient (as the only work-around is try-catch). Maybe we could allow passing default kwarg that would be returned when the passed collection is empty?

[aplavin]

Would make sense for mean as well. Currently, empty mean returns NaN for numbers and throws for non-numbers.

Btw, sum lets you specify the default instead of zero, but the argument is called init there.

[bkamins]

Currently, empty mean returns NaN for numbers and throws for non-numbers.

Yes, but at least it does not throw an error in standard case (but indeed we could allow for providing the other value of default as NaN, while correct is highly non-standard for Julia data processing)

called init there

The reason is that sum is a reduction, while e.g. median is not a reduction, so that is why init is not a best name in this case. But indeed - I was not sure what would be the best name of the kwarg.

[aplavin]

Yes, but at least it does not throw an error in standard case

Would be weird to have default in median, but not in mean. And I also wanted this functionality a couple of times!

NaN, while correct is highly non-standard for Julia data processing

Dunno, I like NaNs the majority of time, it's the only way to signal "there's no number here" in a type-stable way. I mostly wanted to use default for values like 0 or something similar, when I need an actual number.

that is why init is not a best name in this case

Yeah, init doesn't seem to fit well for mean/median, I agree. Just pointed out what exists elsewhere.

[pdeffebach]

FWIW, R returns NULL which is roughly equivalent to nothing.

[aplavin]

For mean, NaN seems the only right default choice, considering that mean(x) = sum(x) / length(x) and 0.0 / 0 is NaN.

However, if these functions all returned nothing for empty inputs, setting a default would be a straightforward something(median(x), 0.0). It's already common for Julia functions to return nothing indicating "no result" in some sense, eg findfirst-like functions.

nothing is also great in that functions deliberately don't propagate it further. So when a user gets nothing and forgets to handle it, he gets an exception very soon.

[bkamins]

These are all good ideas. But then the question is how to convert it to API in a non-breaking way?
We probably need a kwarg that would allow to switch behavior.

[aplavin]

Maybe, median/quantile can simply return nothing instead of throwing? Seems nonbreaking, and allows for easy default selection through something(...).

[aplavin]

Another idea is a general wrapper function so that the interface looks like onempty(median, xs, default=0.0).

[bkamins]

Seems nonbreaking, and allows for easy default selection through something(...).

It would be non-breaking, but I feared proposing it for such a common function 😄.

a general wrapper function so that the interface looks like onempty(median, xs, default=0.0).

This general wrapper function would require special casing for each function passed as its first argument to be efficient.

The point is that isempty(skipmissing(x)) is O(n) operation in general:

julia> x = missings(Int, 10^8);

julia> @time isempty(skipmissing(x))
  0.077613 seconds (1 allocation: 16 bytes)
true

But maybe it is good enough?

[aplavin]

The point is that isempty(skipmissing(x)) is O(n) operation in general

Good point! I mostly considered applying functions to an actual materialized array, not to skipmissing.

An advantage of onempty vs either median(; default) or something(median(), default) is that it makes clear when the default value is used - exactly when the input is empty. The other two variants read more ambiguous regarding when the default gets returned.

OTOH, advantages of something(median(), ...) is that it doesn't involve creating any extra functions, and is trivially efficient wrt isempty(...) checks.

[bkamins]

@nalimilan - what do you think?

[nalimilan]

I don't think there's a precedent of a function that returns nothing when the input is empty, right? So adding a default keyword argument seems simpler. It's too bad we can't call it init for consistency with sum.

[aplavin]

Some Julia functions throw an exception when no result is available (eg first), some do return nothing (eg findfirst).
Btw, first-or-default and only-or-default functions would also useful to have...

[bkamins]

In general where possible I personally prefer special value-based behavior rather than error-based. For median and quantile returning nothing is safe.

However, I understand the concern of @nalimilan that it requires a careful design so that we have one common solution rather than many special cases. E.g. nothing as a special value works for mean, median, quantile, and e.g. std, but it is not a good value for sum as:

julia> isnothing(sum([nothing]))
true

[aplavin]

It's unfortunate, but is a fact, that functions in Julia tend not to really consider that nothing can be contained within a collection. For example, findfirst returns nothing when it didn't find anything, or when it found the result, and its index is nothing:

# can distinguish these two:
julia> findfirst(x -> true, Dict(1 => 2))
1
julia> findfirst(x -> false, Dict(1 => 2))

# but not these two:
julia> findfirst(x -> true, Dict(nothing => 2))

julia> findfirst(x -> false, Dict(nothing => 2))

Potentially, median could reasonably be extended to non-numbers in the future, and return the only element for length-1 collections. But I personally don't see it as a major issue...