group_by() fails when grouping by vctrs and non-vctrs at same time

[dholstius]

Not sure if this is a bug in dplyr or vctrs or my code. Here's a reprex:

library(vctrs)
library(tidyverse)

new_YYYY <- function(x) {
  new_rcrd(
    list(year = x),
    class = "YYYY")
}

test_data <-
  tibble::tibble(
    year = new_YYYY(c(2011, 2011, 2012, 2012)),
    foo = 1:4,
    bar = c("A", "A", "B", "B"))

# Grouping by `year` alone works ...
grouped_by_year <- group_by(test_data, year)

# ... and grouping by `foo` and `bar works (neither are vctrs) ...
grouped_by_foo_and_bar <- group_by(test_data, foo, bar)

# ... but grouping by a combination of `year` (vctrs) and `foo` (numeric) throws an error.
grouped_by_year_and_foo <- group_by(test_data, year, foo)
#> Error: Internal error in `p_equal_na_equal()`: Unsupported vctrs type `dataframe`.

Here's the output of rlang::last_error():

<error/rlang_error>
Internal error in `p_equal_na_equal()`: Unsupported vctrs type `dataframe`.
Backtrace:
 1. dplyr::group_by(test_data, year, foo)
 2. dplyr:::group_by.data.frame(test_data, year, foo)
 3. dplyr::grouped_df(groups$data, groups$group_names, .drop)
 4. dplyr:::compute_groups(data, vars, drop = drop)
 5. dplyr:::vec_split_id_order(group_vars)
 6. vctrs::vec_group_loc(x)
Run `rlang::last_trace()` to see the full context.

... and the output of rlang::last_trace():

<error/rlang_error>
Internal error in `p_equal_na_equal()`: Unsupported vctrs type `dataframe`.
Backtrace:
    █
 1. ├─dplyr::group_by(test_data, year, foo)
 2. └─dplyr:::group_by.data.frame(test_data, year, foo)
 3.   └─dplyr::grouped_df(groups$data, groups$group_names, .drop)
 4.     └─dplyr:::compute_groups(data, vars, drop = drop)
 5.       └─dplyr:::vec_split_id_order(group_vars)
 6.         └─vctrs::vec_group_loc(x)
 7.           └─(function () ...

^{Created on 2021-02-04 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.2 (2019-12-12)
#>  os       macOS Catalina 10.15.7      
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/Los_Angeles         
#>  date     2021-02-04                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date       lib source                            
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 3.6.0)                    
#>  backports     1.1.9   2020-08-24 [1] CRAN (R 3.6.2)                    
#>  blob          1.2.1   2020-01-20 [1] CRAN (R 3.6.0)                    
#>  broom         0.7.0   2020-07-09 [1] CRAN (R 3.6.2)                    
#>  callr         3.5.1   2020-10-13 [1] CRAN (R 3.6.2)                    
#>  cellranger    1.1.0   2016-07-27 [1] CRAN (R 3.6.0)                    
#>  cli           2.3.0   2021-01-31 [1] CRAN (R 3.6.2)                    
#>  colorspace    1.4-1   2019-03-18 [1] CRAN (R 3.6.0)                    
#>  crayon        1.3.4   2017-09-16 [1] CRAN (R 3.6.0)                    
#>  DBI           1.1.0   2019-12-15 [1] CRAN (R 3.6.0)                    
#>  dbplyr        1.4.4   2020-05-27 [1] CRAN (R 3.6.2)                    
#>  desc          1.2.0   2018-05-01 [1] CRAN (R 3.6.0)                    
#>  devtools      2.3.1   2020-07-21 [1] CRAN (R 3.6.2)                    
#>  digest        0.6.27  2020-10-24 [1] CRAN (R 3.6.2)                    
#>  dplyr       * 1.0.4   2021-02-02 [1] CRAN (R 3.6.2)                    
#>  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 3.6.2)                    
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 3.6.0)                    
#>  forcats     * 0.5.0   2020-03-01 [1] CRAN (R 3.6.0)                    
#>  fs            1.4.2   2020-06-30 [1] CRAN (R 3.6.2)                    
#>  generics      0.0.2   2018-11-29 [1] CRAN (R 3.6.0)                    
#>  ggplot2     * 3.3.2   2020-06-19 [1] CRAN (R 3.6.2)                    
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 3.6.2)                    
#>  gtable        0.3.0   2019-03-25 [1] CRAN (R 3.6.0)                    
#>  haven         2.3.1   2020-06-01 [1] CRAN (R 3.6.2)                    
#>  highr         0.8     2019-03-20 [1] CRAN (R 3.6.0)                    
#>  hms           0.5.3   2020-01-08 [1] CRAN (R 3.6.0)                    
#>  htmltools     0.5.0   2020-06-16 [1] CRAN (R 3.6.2)                    
#>  httr          1.4.2   2020-07-20 [1] CRAN (R 3.6.2)                    
#>  jsonlite      1.7.1   2020-09-07 [1] CRAN (R 3.6.2)                    
#>  knitr         1.30    2020-09-22 [1] CRAN (R 3.6.2)                    
#>  lifecycle     0.2.0   2020-03-06 [1] CRAN (R 3.6.0)                    
#>  lubridate     1.7.9   2020-06-08 [1] CRAN (R 3.6.2)                    
#>  magrittr      1.5     2014-11-22 [1] CRAN (R 3.6.0)                    
#>  memoise       1.1.0   2017-04-21 [1] CRAN (R 3.6.0)                    
#>  modelr        0.1.8   2020-05-19 [1] CRAN (R 3.6.2)                    
#>  munsell       0.5.0   2018-06-12 [1] CRAN (R 3.6.0)                    
#>  pillar        1.4.6   2020-07-10 [1] CRAN (R 3.6.2)                    
#>  pkgbuild      1.1.0   2020-07-13 [1] CRAN (R 3.6.2)                    
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 3.6.0)                    
#>  pkgload       1.1.0   2020-05-29 [1] CRAN (R 3.6.2)                    
#>  prettyunits   1.1.1   2020-01-24 [1] CRAN (R 3.6.1)                    
#>  processx      3.4.5   2020-11-30 [1] CRAN (R 3.6.2)                    
#>  ps            1.3.4   2020-08-11 [1] CRAN (R 3.6.2)                    
#>  purrr       * 0.3.4   2020-04-17 [1] CRAN (R 3.6.1)                    
#>  R6            2.4.1   2019-11-12 [1] CRAN (R 3.6.1)                    
#>  Rcpp          1.0.5   2020-07-06 [1] CRAN (R 3.6.2)                    
#>  readr       * 1.3.1   2018-12-21 [1] CRAN (R 3.6.0)                    
#>  readxl        1.3.1   2019-03-13 [1] CRAN (R 3.6.0)                    
#>  remotes       2.2.0   2020-07-21 [1] CRAN (R 3.6.2)                    
#>  reprex        0.3.0   2019-05-16 [1] CRAN (R 3.6.0)                    
#>  rlang         0.4.10  2020-12-30 [1] CRAN (R 3.6.2)                    
#>  rmarkdown     2.4.6   2020-10-20 [1] Github (rstudio/rmarkdown@7239cea)
#>  rprojroot     1.3-2   2018-01-03 [1] CRAN (R 3.6.0)                    
#>  rvest         0.3.6   2020-07-25 [1] CRAN (R 3.6.2)                    
#>  scales        1.1.1   2020-05-11 [1] CRAN (R 3.6.2)                    
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 3.6.0)                    
#>  stringi       1.5.3   2020-09-09 [1] CRAN (R 3.6.2)                    
#>  stringr     * 1.4.0   2019-02-10 [1] CRAN (R 3.6.0)                    
#>  testthat      3.0.1   2020-12-17 [1] CRAN (R 3.6.2)                    
#>  tibble      * 3.0.3   2020-07-10 [1] CRAN (R 3.6.2)                    
#>  tidyr       * 1.1.1   2020-07-31 [1] CRAN (R 3.6.2)                    
#>  tidyselect    1.1.0   2020-05-11 [1] CRAN (R 3.6.2)                    
#>  tidyverse   * 1.3.0   2019-11-21 [1] CRAN (R 3.6.0)                    
#>  usethis       1.6.1   2020-04-29 [1] CRAN (R 3.6.2)                    
#>  vctrs       * 0.3.6   2020-12-17 [1] CRAN (R 3.6.2)                    
#>  withr         2.4.1   2021-01-26 [1] CRAN (R 3.6.2)                    
#>  xfun          0.18    2020-09-29 [1] CRAN (R 3.6.2)                    
#>  xml2          1.3.2   2020-04-23 [1] CRAN (R 3.6.2)                    
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 3.6.0)                    
#> 
#> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Thank you in advance for your time and attention!

[DavisVaughan]

I haven’t confirmed yet, but I’d bet this is a regression introduced by #1256

[DavisVaughan]

Simplest reprex:

library(vctrs)

col <- new_rcrd(list(x = 1))
df <- data_frame(col = col, z = 1)
proxy <- vec_proxy_equal(df)

# data frame print method hides the details
proxy
#>   x z
#> 1 1 1

names(proxy)
#> [1] "col" "z"

# uh oh, didnt get flattened!
proxy$col
#>   x
#> 1 1
proxy$z
#> [1] 1

With a bit more detail, showing that the problem only occurs with rcrds containing 1 field.

library(tibble)
library(vctrs)

df_col1 <- data_frame(x = 1)
df_col2 <- data_frame(x = 1, y = 1)
rcrd_col1 <- new_rcrd(list(x = 1))
rcrd_col2 <- new_rcrd(list(x = 1, y = 2))

df1 <- data_frame(df_col1 = df_col1, z = 1)
df2 <- data_frame(df_col2 = df_col2, z = 1)
df3 <- data_frame(rcrd_col1 = rcrd_col1, z = 1)
df4 <- data_frame(rcrd_col1 = rcrd_col2, z = 1)

# `df_col1` unwraps itself, so this is correct
as_tibble(vec_proxy_equal(df1))
#> # A tibble: 1 x 2
#>   df_col1     z
#>     <dbl> <dbl>
#> 1       1     1

# `df_col2` doesn't unwrap itself because it has 2 columns.
# `df_flatten()` recognizes that the number of columns from flattening would
# be different, so it flattens and we get the correct result.
as_tibble(vec_proxy_equal(df2))
#> # A tibble: 1 x 3
#>       x     y     z
#>   <dbl> <dbl> <dbl>
#> 1     1     1     1

# WRONG
# `rcrd_col1` returns a data frame proxy with 1 column.
# This is NOT unwrapped because we didn't go through `df_proxy()`.
# Calling `df_flatten()` on the outer data frame early exists because it
# looks like the number of columns won't be changed from the flattening,
# and it assumes that any df-cols with 1 col would have been unwrapped.
as_tibble(vec_proxy_equal(df3))
#> # A tibble: 1 x 2
#>   rcrd_col1$x     z
#>         <dbl> <dbl>
#> 1           1     1

# `rcrd_col2` returns a data frame proxy with 2 columns.
# `df_flatten()` recognizes that the number of columns from flattening would
# be different, so it flattens and we get the correct result.
as_tibble(vec_proxy_equal(df4))
#> # A tibble: 1 x 3
#>       x     y     z
#>   <dbl> <dbl> <dbl>
#> 1     1     2     1

I see two paths forward:

1. Add vec_proxy_equal.vctrs_rcrd() that calls df_proxy(new_data_frame(x)) so that 1 field rcrds get unwrapped.
2. Fix df_flatten() to only early exit if the number of columns hasn't changed and we didn't see any data frame columns when counting the flat-width.

I think I am in favor of 2, since df_proxy() is an internal vctrs function and we generally try to keep that separate from the implementations of vctrs-vctr and vctrs-rcrd (since in theory they could live in their own packages). Fixing df_flatten() would also be more robust to the possibility of other package developers returning their own data frame proxies without knowing that they should unwrap 1 column data frames.