NA handling in unite

[voxnonecho]

Consider the following df:

ID   d1   d2   
1    G    G
2    A    G
3    A    A
4    G    A
5    NA   NA
6    G    G

When uniting d1 and d2:

tidyr::unite(df, new, d1, d2, remove = FALSE, sep = "")

Row 5 gives NANA instead of the expected NA

  ID  new   d1   d2
1  1   GG    G    G
2  2   AG    A    G
3  3   AA    A    A
4  4   GA    G    A
5  5 NANA <NA> <NA>
6  6   GG    G    G

[hadley]

unite() is just following the standard paste rules:

paste(NA, NA)
#> [1] "NA NA"

[voxnonecho]

I was thinking of a pre-processing treatement similar to: with(df, ifelse(is.na(d1)|is.na(d2), NA, paste0(d1, d2))).

[hadley]

I think you need a compelling argument as to why unite() should work differently to paste()

[voxnonecho]

Well, I think unite() should work like paste() but could maybe provide an additional argument to handle NAs, à la na.rm = TRUE

[danrlu]

I think in some cases the omit NA option could be useful. My df has many columns that contain mostly NA, as a result of multiple rounds of join.

recipe  potato  tomato  cucumber    rock
A       potato  NA      cucumber    NA
B       NA      NA      NA          rock
C       NA      tomato  NA          NA
...

So I was trying to combine the columns into one and remove the NA to see things better.

recipe  ingredients
A       potato,cucumber
B       rock
C       tomato
...

The solution is not hard, just not quite as tidy.

[tjohnson250]

I just ran into this issue and also suggest adding an option to handle NA in unite. In fact, I'd suggest that the following expressions (though perhaps with an extra param to omit NAs in unite) should produce output identical to its input:

df <- data.frame(x = c("a", "a b", "a b c", NA))
df
x
1 a
2 a b
3 a b c
4
df %>% separate(x, c("a", "b"), extra = "merge") %>% unite(x, a, b, sep=" ")
x
1 a NA
2 a b
3 a b c
4 NA NA
Warning message:
Too few values at 1 locations: 1

In other words, if separate and unite are complements one should be able to use one of them to reverse the operation of the other.

[jennybc]

This is not the requested solution, but a clean way to get the desired result is:

library(tidyverse)
df <- tribble(
  ~ID, ~d1, ~d2,   
    1, "G", "G",
    2, "A", "G",
    3, "A", "A",
    4, "G", "A",
    5,  NA,  NA,
    6, "G", "G")
df %>% 
  replace_na(list(d1 = "", d2 = "")) %>% 
  unite(new, d1, d2, remove = FALSE, sep = "")
#> # A tibble: 6 × 4
#>      ID   new    d1    d2
#> * <dbl> <chr> <chr> <chr>
#> 1     1    GG     G     G
#> 2     2    AG     A     G
#> 3     3    AA     A     A
#> 4     4    GA     G     A
#> 5     5                  
#> 6     6    GG     G     G

[e-linc]

I would also suggest adding an option of na.rm = TRUE to handle NAs. Although @jennybc 's alternative solution works for this particular problem, it will show blanks and separators when the separator is not "".

My problem is the same with @danrlu 's. Is there a better and neat solution to ignore the NAs? Currently I just unite all columns and then str_replace_all NAs and adjacent separators with empty strings.

[alexpghayes]

Here is a generalization of unite that allows you to create a new column from an arbitrary function applied to columns selected as you would with unite. The function needs to return a character vector because it relies on pmap_chr, but swap in pmap_* to taste.

library(tidyverse)

df <- tribble(
  ~ID, ~d1, ~d2, ~d3,
  1, "G", "G", "C",
  2, NA, "G", "T",
  3, "A", NA, "G",
  4, "G", "A", "A",
  5,  NA,  NA, NA,
  6, "G", "G", "G")

#' Semi-general  \code{unite} to vectorize a function across columns of dataframe
#'
#' Accepts columns from a dataframe and vectorizes/parallel maps a function
#' across them, returning the result in a new column. Function must return a
#' character vector because \code{purrr::pmap_char} enforces type-safety.
#'
#' @param df A dataframe
#' @param col Bare (unquoted) name of results column
#' @param ... Bare (unquoted) names of argument columns
#' @param fun A function that accepts as many arguments as provided argument
#' columns. Gets passed to \code{purrr::pmap_chr} so formula-style lambda
#' specification also works.
#' @param remove Whether or not to remove the argument columns (defaults
#' to \code{true})
#'
#' @return Dataframe with new column generated by applying \code{fun} to
#' argument columns element-wise

combine <- function(df, col, ..., fun, remove = TRUE) {
  to_merge <- quos(...)
  new_col <- quo_name(enquo(col))
  merge_cols <- map_chr(to_merge, quo_name)

  df <- mutate(df, !!new_col := pmap_chr(list(!!!to_merge), fun))
  if (remove) df <- select(df, -one_of(merge_cols))
  df
}

combine(df, new, d1, d2, d3, fun = paste0)

#> # A tibble: 6 x 2
#>      ID    new
#>   <dbl>  <chr>
#> 1     1    GGC
#> 2     2   NAGT
#> 3     3   ANAG
#> 4     4    GAA
#> 5     5 NANANA
#> 6     6    GGG

I'm not a huge fan of this interface, but I have found it more useful than unite a couple of times. I've also found myself doing stupid things with this function and feel like it's some sort of anti-pattern.

If there are any changes coming to the unite interface, I'd enjoy something in the vein of facilitating vectorized operations for non-vectorized function across dataframe columns.

[alistaire47]

I'm not convinced that unite should work like paste, as it's a very rare situation when a user would actually want to turn NA values into strings. More concerningly, in terms of API consistency separate will introduce NAs in a way that unite can't reverse:

library(tidyr)

example <- tibble::data_frame(x = c('foo', 'foo bar', 'foo bar baz'))

example %>% separate(x, c('foo', 'bar', 'baz'), fill = 'right')    # without `fill = 'right'` same result with a message 
#> # A tibble: 3 x 3
#>     foo   bar   baz
#> * <chr> <chr> <chr>
#> 1   foo  <NA>  <NA>
#> 2   foo   bar  <NA>
#> 3   foo   bar   baz

example %>% 
    separate(x, c('foo', 'bar', 'baz'), fill = 'right') %>% 
    unite(x, foo:baz, sep = ' ')
#> # A tibble: 3 x 1
#>             x
#> *       <chr>
#> 1   foo NA NA
#> 2  foo bar NA
#> 3 foo bar baz

If NAs are in the middle of columns that get united and then separated then paste-like behavior would allow the NA location to be saved (at the cost of requiring them to be converted from strings to actual NA again), but most of the time the NA handling keeps the functions from being inverses. Making na.rm = TRUE the default would be a breaking change, but probably not one that would break much code.

[hadley]

There are actually two feature requests in this thread:

1. Make NAs infections so that if any input is NA, then the output is NA
2. Provide an easy way to drop NAs.

2. seems like the more useful option so I will implement that.

@alexpghayes the plan is to extract out a general helper for turning the vectorised functions that power many tidyr functions in a tibblicious version