[C++] joins segfault when data contains list column

[asfimport]

When I run the R code below, it results in a segfault if one of the tables contains a list column.

library(arrow)
library(dplyr)

basic_tbl <- arrow_table(
  tibble::tibble(
    x = 1:3,
    y = c("a", "b", "c")
  )
)

basic_tbl2 <- arrow_table(
  tibble::tibble(
    x = 1:3,
    z = c(T, F, T)
  )
)

list_tbl <- arrow_table(
  tibble::tibble(
    z = list(c("first", "list", "col", "row"), c("second row ", "here")),
    x = 1:2
  )
)

# works
left_join(basic_tbl, basic_tbl2) %>%
  collect()

# segfaults
left_join(basic_tbl, list_tbl) %>%
  collect()

Reporter: Nicola Crane / @thisisnic
Assignee: David Li / @lidavidm

Related issues:

• [C++] Enable joins when data contains a list column (is related to)

PRs and other links:

• GitHub Pull Request #11625

_{Note: This issue was originally created as ARROW-14519. Please see the migration documentation for further details.}

[asfimport]

Jonathan Keane / @jonkeane:
It looks like it even segfaults when specifying the join keys with by = "x" (so it's not only that that list column is trying to be used as a join key!

[asfimport]

David Li / @lidavidm:
This is because joining on lists is not supported, but the code path triggers an assertion instead of reporting an error. Also, it looks like the join code needs to pre-process all columns, so the presence of any unsupported type will cause this (as you found). We should at least raise an error instead of crashing, but I'm not familiar enough with the join code to know if we can handle unsupported types when they're not being used as the key.

[asfimport]

Neal Richardson / @nealrichardson:
Sounds related to ARROW-14181 yeah?

[asfimport]

David Li / @lidavidm:
ARROW-14181 will expand the set of supported types but won't affect things here. (Also, the code path in question here already supported dictionaries.)

[asfimport]

Michal Nowakiewicz / @michalursa:
We cannot easily support more types in hash join right now. That is because we transform and encode all the input values, key and non-key (row_encoder.h), so it would need another specialization for each additional type.

But we can return an error (from HashJoinSchema::ValidateSchemas where we check data types from input schemas and keys) instead of asserting.

[asfimport]

Neal Richardson / @nealrichardson:
I don't follow, why do we have to transform and encode columns that we aren't joining by? They're just along for the ride.

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 11625
#11625