$join() with join_nulls = TRUE doesn't produce expected result with DataFrames

[george-wood]

I was using the join_nulls argument in $join()$, added in #716. Joining LazyFrames with join_nulls = TRUE produces the expected result. However, join_nulls seems to be ignored when joining DataFrames. I'd be happy to look into this if it might be a good first issue?

library(polars)

df1 <-
  pl$DataFrame(
    x = "John",
    y = NULL
  )

df2 <-
  pl$DataFrame(
    x = "John",
    y = NULL,
    value = 1
  )

df1$
  join(
    other = df2,
    on = c("x", "y"),
    how = "left",
    join_nulls = TRUE
  )
#> shape: (1, 3)
#> ┌──────┬──────┬───────┐
#> │ x    ┆ y    ┆ value │
#> │ ---  ┆ ---  ┆ ---   │
#> │ str  ┆ null ┆ f64   │
#> ╞══════╪══════╪═══════╡
#> │ John ┆ null ┆ null  │
#> └──────┴──────┴───────┘

# lazy
df1$lazy()$
  join(
    other = df2$lazy(),
    on = c("x", "y"),
    how = "left",
    join_nulls = TRUE
  )$
  collect()
#> shape: (1, 3)
#> ┌──────┬──────┬───────┐
#> │ x    ┆ y    ┆ value │
#> │ ---  ┆ ---  ┆ ---   │
#> │ str  ┆ null ┆ f64   │
#> ╞══════╪══════╪═══════╡
#> │ John ┆ null ┆ 1.0   │
#> └──────┴──────┴───────┘

^{Created on 2024-03-19 with reprex v2.1.0}

There is a test for join_nulls = TRUE using LazyFrames, but not DataFrames:

r-polars/tests/testthat/test-joins.R

Line 173 in 0f98aa2

test_that("argument 'join_nulls' works", {

[eitsupi]

Thanks for reporting!
Since this could be an upstream (Rust) issue, could you please check if you can reproduce it in Python if possible?

If it does not reproduce in Python, it is a problem with this repository and I would be happy for you to work on it; if you get the same result in Python, we will have to report the problem upstream.

[george-wood]

Thanks! I checked in Python and it returned the same (expected) output for LazyFrames and DataFrames, so it seems to be a problem with r-polars.

import polars as pl

df1 = pl.DataFrame(
  {
    "x": ["John"],
    "y": [None]
  }
)

df2 = pl.DataFrame(
  {
    "x": ["John"],
    "y": [None],
    "value": [1]
  }
)

print(
  df1
  .join(
    other = df2,
    how = "left", 
    on = ["x", "y"],
    join_nulls = True
  )
)
#> shape: (1, 3)
#> ┌──────┬──────┬───────┐
#> │ x    ┆ y    ┆ value │
#> │ ---  ┆ ---  ┆ ---   │
#> │ str  ┆ null ┆ i64   │
#> ╞══════╪══════╪═══════╡
#> │ John ┆ null ┆ 1     │
#> └──────┴──────┴───────┘

# lazy
print(
  df1.lazy()
  .join(
    other = df2.lazy(),
    how = "left", 
    on = ["x", "y"],
    join_nulls = True
  )
  .collect()
)
#> shape: (1, 3)
#> ┌──────┬──────┬───────┐
#> │ x    ┆ y    ┆ value │
#> │ ---  ┆ ---  ┆ ---   │
#> │ str  ┆ null ┆ i64   │
#> ╞══════╪══════╪═══════╡
#> │ John ┆ null ┆ 1     │
#> └──────┴──────┴───────┘

^{Created at 2024-03-19 16:43:14 GMT by reprexlite v0.5.0}

[etiennebacher]

The bug is actually very simple: the user can provide the join_nulls arg but it is never passed to the lazy method:

r-polars/R/dataframe__frame.R

Lines 1008 to 1029 in d30005b

	DataFrame_join = function(
	other,
	on = NULL,
	how = c("inner", "left", "outer", "semi", "anti", "cross", "outer_coalesce"),
	...,
	left_on = NULL,
	right_on = NULL,
	suffix = "_right",
	validate = "m:m",
	join_nulls = FALSE,
	allow_parallel = TRUE,
	force_parallel = FALSE) {
	if (!is_polars_df(other)) {
	Err_plain("`other` must be a DataFrame.") |>
	unwrap("in $join():")
	}
	.pr$DataFrame$lazy(self)$join(
	other = other$lazy(), left_on = left_on, right_on = right_on,
	on = on, how = how, suffix = suffix, allow_parallel = allow_parallel,
	force_parallel = force_parallel
	)$collect()
	}

@george-wood I think this is a good first issue, do you want to fix this and add a test? Feel free to open an incomplete PR if you struggle somewhere and we can see how to carry it to the finish line

[george-wood]

@etiennebacher I'd be happy to.

[george-wood]

I noticed that the validate arg is also not passed to the lazy method. I'll fix this too.