Add functionalities to deal with missing

[DominiqueMakowski]

I've been suprised that we don't have a function to remove all cols that have at least one missing?

like data_complete() or data_remove_missing()

Fairly low maintenance cost for quite a useful convenience feature?

This function could potentially have an argument (e.g., report=TRUE) to return a return a report, for each observation, of how many missing cols and in what cols?

[etiennebacher]

data_complete() would be confusing with tidyr::complete() that creates all combinations of values.

I wonder what's the best way to combine this with our existing remove_empty_*() functions.

Just wondering, does it happen often to you to drop entire columns because they have one missing value? I understand dropping rows if for instance the age is missing, but I don't remember seeing this with columns.

---

Quick implementation if we go forward with this:

drop_cols_with_missing <- function(x) {
  # input checks...
  x[] <- lapply(x, function(col) {
    if (anyNA(col)) {
      return(NULL)
    } else {
      col
    }
  })
  x
}

dat <- data.frame(x = 1:3, y = NA, z = c(1, 2, NA))

drop_cols_with_missing(dat)
#>   x
#> 1 1
#> 2 2
#> 3 3

[strengejacke]

Do you have a use case for such a function? We have remove_na arguments in many functions to remove missings from the output, and - as Etienne wrote - functions to remove "empty" variables or cases.

[DominiqueMakowski]

Yes, one usecase example would be to make explicit and across-the-board dealing with missing decisions, in workflows such as:

data |> 
  data_select(vars_of_interest) |> 
  remove_missing() |> 
  correlation::correlation()

[strengejacke]

Why would you need to remove missing values? Here's an example where both variables have missing values, which are (case-wise) deleted automatically:

data(efc, package = "datawizard")
efc$e42dep <- as.numeric(efc$e42dep)

efc |> 
  datawizard::data_select(c("e42dep", "c172code")) |> 
  correlation::correlation()
#> # Correlation Matrix (pearson-method)
#> 
#> Parameter1 | Parameter2 |    r |        95% CI | t(86) |     p
#> --------------------------------------------------------------
#> e42dep     |   c172code | 0.05 | [-0.16, 0.26] |  0.49 | 0.627
#> 
#> p-value adjustment method: Holm (1979)
#> Observations: 88

^{Created on 2025-04-02 with reprex v2.1.1}

[mattansb]

(They're actually pair-wise deleted in {correlation})

I also don't understand the utility of removing columns with missing values.

[DominiqueMakowski]

They're actually pair-wise deleted

I know, but in SOME cases you might want to keep the same observations for all correlations. And Andy (who might be in this case) suggested adding an argument on how to handle missing inside correlation(). But I said, if the users wants something else than the default (pairwise deletion) I think they should explicitly deal with missing outside of correlation().

Hence how the issue came about lol

[mattansb]

Ah I see - then perhaps we should add that directly as an argument to correlation()¹, similar to stats:cor(use=).

Footnotes

1. Maybe it's time someone finishes working on {correlation2} 😔 ↩

[mattansb]

Added here easystats/correlation#346

[etiennebacher]

Seems like easystats/correlation#346 solves this issue, so closing. Feel free to reopen if I misunderstood.