joins match rows on NA across datasets

[garrettgman]

Below, inner_join() matches two rows that each have an NA for the key. This seems to violate the intuition about NA's (and leads to an incorrect result).

This seems to happen for all of the joins.

library(dplyr)
​
songs <- data_frame(song = c("Do-Re-Mi", "A Spoonful of Sugar"), movie = c("The Sound of Music", NA))
songs
##                  song              movie
##                 <chr>              <chr>
## 1            Do-Re-Mi The Sound of Music
## 2 A Spoonful of Sugar               <NA>
​
singers <- data_frame(movie = c(NA, "The Sound of Music"), singer = c("Arnold Schwarzenegger", "Julie Andrews"))
singers
##                movie                singer
##                <chr>                 <chr>
## 1               <NA> Arnold Schwarzenegger
## 2 The Sound of Music         Julie Andrews
​
songs %>% inner_join(singers, by = "movie")
##                  song              movie                singer
##                 <chr>              <chr>                 <chr>
## 1            Do-Re-Mi The Sound of Music         Julie Andrews
## 2 A Spoonful of Sugar               <NA> Arnold Schwarzenegger

[krlmlr]

Thanks. This is clearly something that SQL does differently:

memdb_frame(a=c(1, NA), b = 1:2) %>% left_join(memdb_frame(a = c(1, NA), c = 2:3))

@hadley: Can we safely change the NA behavior for joins?

[hadley]

Yeah, an NA should never match another NA.

I have a vague memory that this has come up in the past, and some issue may have yielded the current behaviour.

[krlmlr]

See also examples in #2272.

[krlmlr]

This is fixable, but getting it right for all data types in all combinations will take some time.

[krlmlr]

@hadley: Should NaN match another NaN? We have is.na(NaN) in R, but current dplyr code considers NA_real_ != NaN but NA_real_ == NA_real_ and NaN == NaN.

[krlmlr]

Joins on NA strings were deliberate at some point (#892), and are tested explicitly. This might be a breaking change unless we introduce some kind of compatibility argument. I still think we shouldn't treat NA as equal for joining.

[hadley]

I'd say NaN shouldn't match anything (like NA), and the string issue must've been= confusion on my part.