NA handling in slice_min / slice_max

[kmillar]

slice_min and slice_max drop rows where the value in the order_by column is NA, returning fewer than the requested number of rows if n is large enough.

It's unclear to me whether this is a bug or just intended behavior that requires documenting.

library(tibble)
library(dplyr)

x <- tibble(a = 1:3, b = c(10, NA, 12))
slice_min(x, b, n = 3)

[hadley]

Reprex:

library(dplyr, warn.conflicts = FALSE)

df <- tibble(a = 1:3, b = c(10, NA, 12))
df |> slice_min(b, n = 3)
#> # A tibble: 2 × 2
#>       a     b
#>   <int> <dbl>
#> 1     1    10
#> 2     3    12

^{Created on 2022-04-15 by the reprex package (v2.0.1)}

I think this is just a documentation issue as an NA value can never be the smallest or largest value.

[hadley]

Combining with #6347:

Currently we have an inconsistency depending on the value of with_ties:

library(dplyr, warn.conflicts = FALSE)

df <- tribble(
  ~id, ~x,
  1, NA,
  1, NA,
  2, 1,
  2, NA,
  3, 1,
  3, 1,
  3, NA
)

df |> group_by(id) |> slice_min(x, n = 1, with_ties = TRUE)
#> # A tibble: 3 × 2
#> # Groups:   id [2]
#>      id     x
#>   <dbl> <dbl>
#> 1     2     1
#> 2     3     1
#> 3     3     1
df |> group_by(id) |> slice_min(x, n = 1, with_ties = FALSE)
#> # A tibble: 3 × 2
#> # Groups:   id [3]
#>      id     x
#>   <dbl> <dbl>
#> 1     1    NA
#> 2     2     1
#> 3     3     1

^{Created on 2022-07-21 by the reprex package (v2.0.1)}

What should happen when there are some NAs in a group? What should happen if there are only NAs in a group?

[hadley]

At a minimum, it feels like we also need an na.rm argument here.

[hadley]

If we think about these functions strictly, if there's even a single NA we can't actually know what row contains the actually the minimum or maximum value, and this would imply that we should always show NAs first, before the known values.

But that behaviour seems pretty annoying/useless, and it would be inconsistent with arrange() which always puts NA values at the end (regardless of whether you're performing an ascending or descending sort). So I think by default we should use missing values to "fill in" the number of requested rows if they're available, and otherwise provide a na.rm option to remove rows containing NAs from the output.

@eutwt @kmillar does this reasoning make sense to you?

[eutwt]

That makes sense to me. So the output of df %>% slice_min(x, n = slice_n) would be the same as df %>% arrange(x) %>% slice_head(n = slice_n) except possibly with extra rows at the end when there are ties in x.

The next question is, are rows with a missing value considered ties? i.e., what should this return?

df <- tibble(x = c(1, NA, NA))

df %>% slice_min(x, n = 2)

[hadley]

Yeah, I think we'd consider NA to be tied with NA in that case.

I think these are the main cases:

# Sufficient unique values; get n rows in output
df <- tibble(x = c(1, 2, 3, NA, NA))
df %>% slice_min(x, n = 2)

# Insufficient values; get fewer than n rows in output
df <- tibble(x = c(1))
df %>% slice_min(x, n = 2)

# Insufficient non-missing values, get NAs in output
df <- tibble(x = c(1, NA))
df %>% slice_min(x, n = 2, na.rm = TRUE)

# Ties; get more than n rows in output
df <- tibble(x = c(1, 2, 2))
df %>% slice_min(x, n = 2)

df <- tibble(x = c(1, NA, NA))
df %>% slice_min(x, n = 2)

# Ties and insufficient values happen cancel out
df <- tibble(x = c(1, 1))
df %>% slice_min(x, n = 2)

[eutwt]

Edited this because I think I initially misunderstood what you meant

# Insufficient values; get fewer than n rows in output
df <- tibble(x = c(1))
df %>% slice_min(x, n = 2)

# Insufficient non-missing values, get NAs in output
df <- tibble(x = c(1, NA))
df %>% slice_min(x, n = 2, na.rm = TRUE)

I was thinking that na.rm = TRUE would behave as-if you removed the NA rows before the slice_m..() step and then it behaved as usual, like adding a filter before the slice. So the group size when na.rm = TRUE would be the number of rows with non-missing values, and then the usual truncate at group-size rule applies using that size. I assume na.rm = FALSE would be the default, so that the only time you get an output smaller than the input despite a large-enough n is when you've actively chosen a special option.

If NAs come last in sorting, the behavior you've given for na.rm makes the na.rm argument equivalent to choosing whether you want to treat NAs as tied. So na.rm = TRUE means "don't treat NAs as tied" because it only has an effect when there would otherwise be more than the supplied n rows due to ties. In that case maybe the argument should be called tied_missing and tied_missing = FALSE would correspond to the behavior you gave for na.rm = TRUE. I think it would be unintuitive to get some NA values back after setting na.rm = TRUE.

[hadley]

Ooops, I think I messed that up. I meant:

# Insufficient values; get fewer than n rows in output
df <- tibble(x = c(1))
df %>% slice_min(x, n = 2)

# Insufficient non-missing values, get fewer than n rows in output
df <- tibble(x = c(1, NA))
df %>% slice_min(x, n = 2, na.rm = TRUE)

# Insufficient non-missing values, get NAs in output
df <- tibble(x = c(1, NA))
df %>% slice_min(x, n = 2)

[eutwt]

Ha! Yes that makes sense now