tidyr::unnest_wider casting error when unnesting group with all NULL in first submission

[florianm]

Problem

tidyr::unnest_wider trips over its own feet when trying to unnest a key, here photos_turtle, where all elements (all four possible photos) are NULL in the first list item.

Should ruODK maybe parse the form schema, unnest each existing column explicitly, and cast it into an appropriate type? This would take the tidyr::unnest_wider magic out of the picture entirely.

Reproducible example

  suppressMessages(library(tidyverse))
  library(ruODK)
  prod <- "https://odkcentral.dbca.wa.gov.au"
  uat <- "https://odkcentral-uat.dbca.wa.gov.au"

  ruODK::ru_setup(pid=1, fid="build_Marine-Wildlife-Incident-0-6_1559789189", url=prod)
  ft <- ruODK::odata_service_get()
  mwi_prod <- ruODK::odata_submission_get(table = ft$url[[1]], verbose = T, wkt=T)

The offending item is the first list element containing a nested list containing all NULLs.

The same error exists with other objects where the first record contains NULL values.

Error

<error/vctrs_error_incompatible_cast>
Can't cast `photos_turtle$...1` <logical> to `photos_turtle$...1` <vctrs_unspecified>.
Backtrace:
     █
  1. ├─mwi_prod %>% ruODK::odata_submission_parse()
  2. │ ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  3. │ └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  4. │   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  5. │     └─`_fseq`(`_lhs`)
  6. │       └─magrittr::freduce(value, `_function_list`)
  7. │         ├─base::withVisible(function_list[[k]](value))
  8. │         └─function_list[[k]](value)
  9. │           └─ruODK::odata_submission_parse(.)
 10. │             └─`%>%`(...) /home/florian/projects/ruODK/R/odata_submission_parse.R:98:2
 11. │               ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
 12. │               └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 13. │                 └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 14. │                   └─ruODK:::`_fseq`(`_lhs`)
 15. │                     └─magrittr::freduce(value, `_function_list`)
 16. │                       └─function_list[[i]](value)
 17. │                         └─ruODK::unnest_all(., names_repair = names_repair, verbose = verbose)
 18. │                           └─ruODK::unnest_all(nested_tbl, names_repair = names_repair, verbose = verbose) /home/florian/projects/ruODK/R/odata_submission_parse.R:59:4
 19. │                             ├─base::suppressMessages(...) /home/florian/projects/ruODK/R/odata_submission_parse.R:46:6
 20. │                             │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
 21. │                             └─tidyr::unnest_wider(nested_tbl, colname, names_repair = names_repair) /home/florian/projects/ruODK/R/odata_submission_parse.R:46:6
 22. │                               └─tidyr::unchop(data, !!col, keep_empty = TRUE)
 23. │                                 └─vctrs::vec_rbind(!!!x, .ptype = ptype)
 24. ├─vctrs:::vec_cast_dispatch(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 25. ├─vctrs::vec_cast.data.frame(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 26. ├─vctrs:::vec_cast.data.frame.data.frame(...)
 27. ├─vctrs:::vec_cast_dispatch(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 28. └─vctrs:::vec_cast.default(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 29.   └─vctrs::stop_incompatible_cast(x, to, x_arg = x_arg, to_arg = to_arg)
 30.     └─vctrs:::stop_incompatible(...)
 31.       └─vctrs:::stop_vctrs(...)
> 

Solution

Each form should have a first submission with completely filled in fields, which serve as tent poles to help tidyr::unnest_wider do its job. Who knew parsing structured data was that messy, huh.

Session Info

> utils::sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.10

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C               LC_TIME=en_AU.UTF-8       
 [4] LC_COLLATE=en_AU.UTF-8     LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.4.0    stringr_1.4.0    dplyr_0.8.3      purrr_0.3.3.9000 readr_1.3.1     
 [6] tidyr_1.0.0      tibble_2.1.3     ggplot2_3.2.1    tidyverse_1.3.0  ruODK_0.6.6.9004

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5     xfun_0.11            janitor_1.2.0        haven_2.2.0         
 [5] lattice_0.20-38      snakecase_0.11.0     colorspace_1.4-1     vctrs_0.2.0.9007    
 [9] generics_0.0.2       htmltools_0.4.0      rlang_0.4.2.9000     pillar_1.4.2        
[13] glue_1.3.1           withr_2.1.2          DBI_1.0.0            dbplyr_1.4.2        
[17] modelr_0.1.5         readxl_1.3.1         lifecycle_0.1.0.9000 munsell_0.5.0       
[21] gtable_0.3.0         cellranger_1.1.0     rvest_0.3.5          htmlwidgets_1.5.1   
[25] evaluate_0.14        knitr_1.26           curl_4.2             highr_0.8           
[29] broom_0.5.2          Rcpp_1.0.3           backports_1.1.5      scales_1.1.0        
[33] jsonlite_1.6         fs_1.3.1             hms_0.5.2            packrat_0.5.0       
[37] digest_0.6.23        stringi_1.4.3        rlist_0.4.6.1        grid_3.6.1          
[41] cli_1.1.0            tools_3.6.1          magrittr_1.5         lazyeval_0.2.2      
[45] crayon_1.3.4         pkgconfig_2.0.3      ellipsis_0.3.0       data.table_1.12.6   
[49] xml2_1.2.2           reprex_0.3.0         lubridate_1.7.4      rmarkdown_1.18      
[53] assertthat_0.2.1     httr_1.4.1           rstudioapi_0.10      R6_2.4.1            
[57] nlme_3.1-141         compiler_3.6.1 

[florianm]

This is a critical feature for ropensci/software-review#335 - ruODK test cases work fine, but real world data with gaps can trip tidyr.

This bug seems related to tidyverse/tidyr#650 and tidyverse/tidyr#723

Tidyr == 1.0.0.9000 seems to fix this problem. This is too good to be true, need to dig deeper but for now I'm calling it a day.