Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unnest_wider() fails when list columns have additional classes and missing values #1327

Closed
allenbaron opened this issue Feb 25, 2022 · 8 comments · Fixed by #1452
Closed
Labels
feature a feature request or enhancement nesting 🐦 nesting, chopping, and packing
Milestone

Comments

@allenbaron
Copy link

I recently upgraded from 1.1.4 to 1.2.0 and ran into an error with unnest_wider(). It now fails with a type conversion error when two occurrences overlap:

  1. the list column to be unnested is composed of list(s) with at least one empty object (NULL or list()).
  2. the list column to be unnested is composed of list(s) with an additional class.
library(tidyr)
#> Warning: package 'tidyr' was built under R version 4.1.2
library(tibble)

x <- list(a = 1, b = NULL)
df <- tibble::tibble(c = 1, d = list(x))
str(df)
#> tibble [1 × 2] (S3: tbl_df/tbl/data.frame)
#>  $ c: num 1
#>  $ d:List of 1
#>   ..$ :List of 2
#>   .. ..$ a: num 1
#>   .. ..$ b: NULL


# works
tidyr::unnest_wider(df, d)
#> # A tibble: 1 × 3
#>       c     a b    
#>   <dbl> <dbl> <lgl>
#> 1     1     1 NA


# does not work
class(x) <- c("fake", "list")
df <- tibble::tibble(c = 1, d = list(x))
tidyr::unnest_wider(df, d)
#> Error: Can't convert <list> to <fake>.


# works again
x$b <- 1
df <- tibble::tibble(c = 1, d = list(x))
tidyr::unnest_wider(df, d)
#> # A tibble: 1 × 3
#>       c     a     b
#>   <dbl> <dbl> <dbl>
#> 1     1     1     1

Created on 2022-02-25 by the reprex package (v2.0.1)

rlang::last_error()
rlang::last_error()
#> <error/vctrs_error_incompatible_type>
#> Can't convert <list> to <fake>.
#> Backtrace:
#>   1. tidyr::unnest_wider(df, d)
#>   2. tidyr:::df_simplify(...)
#>   3. tidyr:::col_simplify(...)
#>   4. tidyr:::list_init_empty(x, null = TRUE, typed = TRUE)
#>   5. vctrs::vec_assign(x, empty_null, replacement)
#>   7. vctrs::vec_default_cast(...)
#>   8. vctrs::stop_incompatible_cast(...)
#>   9. vctrs::stop_incompatible_type(...)
#>  10. vctrs:::stop_incompatible(...)
#>  11. vctrs:::stop_vctrs(...)
#> Run `rlang::last_trace()` to see the full context.
rlang::last_trace()
rlang::last_trace()
#> <error/vctrs_error_incompatible_type>
#> Can't convert <list> to <fake>.
#> Backtrace:
#>      █
#>   1. └─tidyr::unnest_wider(df, d)
#>   2.   └─tidyr:::df_simplify(...)
#>   3.     └─tidyr:::col_simplify(...)
#>   4.       └─tidyr:::list_init_empty(x, null = TRUE, typed = TRUE)
#>   5.         └─vctrs::vec_assign(x, empty_null, replacement)
#>   6.           └─(function () ...
#>   7.             └─vctrs::vec_default_cast(...)
#>   8.               └─vctrs::stop_incompatible_cast(...)
#>   9.                 └─vctrs::stop_incompatible_type(...)
#>  10.                   └─vctrs:::stop_incompatible(...)
#>  11.                     └─vctrs:::stop_vctrs(...)
Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.1.1 (2021-08-10)
#>  os       macOS Big Sur 10.16         
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2022-02-25                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  ! package     * version date       lib source        
#>    assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.1.0)
#>    backports     1.2.1   2020-12-09 [1] CRAN (R 4.1.0)
#>  P cli           3.0.1   2021-07-17 [?] CRAN (R 4.1.0)
#>    crayon        1.4.1   2021-02-08 [1] CRAN (R 4.1.0)
#>    DBI           1.1.1   2021-01-15 [1] CRAN (R 4.1.0)
#>    digest        0.6.27  2020-10-24 [1] CRAN (R 4.1.0)
#>  P dplyr         1.0.7   2021-06-18 [?] CRAN (R 4.1.0)
#>    ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.1.0)
#>    evaluate      0.14    2019-05-28 [1] CRAN (R 4.1.0)
#>    fansi         0.5.0   2021-05-25 [1] CRAN (R 4.1.0)
#>    fs            1.5.0   2020-07-31 [1] CRAN (R 4.1.0)
#>    generics      0.1.0   2020-10-31 [1] CRAN (R 4.1.0)
#>    glue          1.4.2   2020-08-27 [1] CRAN (R 4.1.0)
#>    highr         0.9     2021-04-16 [1] CRAN (R 4.1.0)
#>    htmltools     0.5.1.1 2021-01-22 [1] CRAN (R 4.1.0)
#>    knitr         1.33    2021-04-24 [1] CRAN (R 4.1.0)
#>    lifecycle     1.0.0   2021-02-15 [1] CRAN (R 4.1.0)
#>    magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.1.0)
#>  P pillar        1.6.2   2021-07-29 [?] CRAN (R 4.1.0)
#>    pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.1.0)
#>    purrr         0.3.4   2020-04-17 [1] CRAN (R 4.1.0)
#>  P R6            2.5.1   2021-08-19 [?] CRAN (R 4.1.0)
#>    reprex        2.0.1   2021-08-05 [1] CRAN (R 4.1.0)
#>    rlang         0.4.11  2021-04-30 [1] CRAN (R 4.1.0)
#>    rmarkdown     2.10    2021-08-06 [1] CRAN (R 4.1.0)
#>    rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.1.0)
#>    sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.1.0)
#>    stringi       1.7.3   2021-07-16 [1] CRAN (R 4.1.0)
#>    stringr       1.4.0   2019-02-10 [1] CRAN (R 4.1.0)
#>    styler        1.5.1   2021-07-13 [1] CRAN (R 4.1.0)
#>    tibble      * 3.1.3   2021-07-23 [1] CRAN (R 4.1.0)
#>    tidyr       * 1.2.0   2022-02-01 [1] CRAN (R 4.1.2)
#>    tidyselect    1.1.1   2021-04-30 [1] CRAN (R 4.1.0)
#>  P utf8          1.2.2   2021-07-24 [?] CRAN (R 4.1.0)
#>    vctrs         0.3.8   2021-04-29 [1] CRAN (R 4.1.0)
#>    withr         2.4.2   2021-04-18 [1] CRAN (R 4.1.0)
#>    xfun          0.25    2021-08-06 [1] CRAN (R 4.1.0)
#>    yaml          2.2.1   2020-02-01 [1] CRAN (R 4.1.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.1/Resources/library
#> 
#>  P ── Loaded and on-disk path mismatch.
@DavisVaughan
Copy link
Member

Were you using some package that created a "fake" object? Or is this something you created yourself?

@DavisVaughan
Copy link
Member

Developer note:

This is due to this section of list_init_empty():

tidyr/R/utils.R

Lines 163 to 166 in 5da45d0

replacement <- list(unspecified(1L))
}
x <- vec_assign(x, empty_null, replacement)

We try to construct a replacement element for all of the NULL elements in the list, replacing them with their size 1 equivalents, i.e. unspecified(1). To use vec_assign() we have to wrap this in a list(), so it is the same type as x. But x here is a classed list, not just a bare list, and you don't automatically get casting methods from a bare list up to a classed list, so this doesn't work out of the box.

Ideally we'd be able to construct a replacement using the classed list, approximately like fake(unspecified(1)), but I'm not sure if that is viable in all cases.

It might be that this is the "best" we can do, and that developer who created "fake" should provide a list->fake cast method.

@allenbaron
Copy link
Author

allenbaron commented Feb 25, 2022

I just created "fake" as a dummy class for the reprex (it shouldn't have any methods). The actual usage where I discovered this was in converting the esummary class of the rentrez package to a data frame. I've already created a work-around for my use case by resetting the esummary class to a list.

@DavisVaughan
Copy link
Member

DavisVaughan commented Feb 25, 2022

Can you please also post a full reprex with rentrez to ensure that we fix that specific problem too? I appreciate the minimal reprex, that is definitely the right thing to do here, but it'll also be nice to know that we fixed this for rentrez too, which might be more complex

@allenbaron
Copy link
Author

library(rentrez)
library(tidyr)
#> Warning: package 'tidyr' was built under R version 4.1.2
library(tibble)

pmid <- "30407550"

x <- rentrez::entrez_summary(
    db = "pubmed",
    pmid,
    always_return_list = TRUE
)

df <- tibble::tibble(id = pmid, record = x)

tidyr::unnest_wider(df, record)
#> Error: Can't convert <list> to <esummary>.

Created on 2022-02-25 by the reprex package (v2.0.1)

allenbaron added a commit to DiseaseOntology/DO.utils that referenced this issue Feb 25, 2022
PROBLEM:
tidyr::unnest_wider() fails with a type conversion error after
upgrading from tidyr v1.1.4 to v1.2.0.

See tidyverse/tidyr#1327.

FIX:
Strip unneeded esummary class before formatting.
allenbaron added a commit to DiseaseOntology/DO.utils that referenced this issue Mar 5, 2022
The newest version of tidyr introduced at least one breaking
change by relying on the strict typing provided by the vctrs
package.

See tidyverse/tidyr#1327
@renzheyu

This comment was marked as off-topic.

@renzheyu

This comment was marked as off-topic.

@hadley hadley added the nesting 🐦 nesting, chopping, and packing label Oct 19, 2022
@hadley

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature a feature request or enhancement nesting 🐦 nesting, chopping, and packing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants