Object not found when function inspects call with a named object made by drake

[Rekyt]

Hey,
Thanks again @wlandau for building drake it is an amazing tool ;)

I'm experiencing a problem with the ade4 package that is due to the way the function niche.param() works (see the reprex).

Basically the function inspects the object call when it was created (call when building nic1), and it fails because it can't find the original object (in my case the dudi1 object). Even though it is perfectly available through drake.

library("drake")
library("ade4")

# Load data beforehand
data("doubs")

# Make first objects (this plan alone works)
my_plan = drake_plan(
  dudi1 = dudi.pca(doubs$env, scale = TRUE, scan = FALSE, nf = 3),
  nic1 = niche(dudi1, doubs$fish, scann = FALSE)
)

make(my_plan)
#> target dudi1
#> target nic1

# Extract parameters (other plan using 'niche.param()')
other_plan = drake_plan(
  omi = niche.param(nic1)
)
# This plan fails
make(rbind(my_plan, other_plan))
#> target omi
#> fail omi
#> Error: Target `omi` failed. Call `diagnose(omi)` for details. Error message:
#>   objet 'dudi1' introuvable

^{Created on 2018-11-30 by the reprex package (v0.2.1)}

Session info

devtools::session_info()
#> - Session info ----------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  os       Windows 8.1 x64             
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  French_France.1252          
#>  ctype    French_France.1252          
#>  tz       Europe/Paris                
#>  date     2018-11-30                  
#> 
#> - Packages --------------------------------------------------------------
#>  package     * version    date       lib source                          
#>  ade4        * 1.7-13     2018-08-31 [1] CRAN (R 3.5.1)                  
#>  assertthat    0.2.0      2017-04-11 [1] CRAN (R 3.5.0)                  
#>  backports     1.1.2      2017-12-13 [1] CRAN (R 3.5.0)                  
#>  base64enc     0.1-3      2015-07-28 [1] CRAN (R 3.5.0)                  
#>  bindr         0.1.1      2018-03-13 [1] CRAN (R 3.5.0)                  
#>  bindrcpp    * 0.2.2      2018-03-29 [1] CRAN (R 3.5.0)                  
#>  callr         3.0.0      2018-08-24 [1] CRAN (R 3.5.1)                  
#>  cli           1.0.1      2018-09-25 [1] CRAN (R 3.5.1)                  
#>  codetools     0.2-15     2016-10-05 [1] CRAN (R 3.5.1)                  
#>  crayon        1.3.4      2017-09-16 [1] CRAN (R 3.5.0)                  
#>  debugme       1.1.0      2017-10-22 [1] CRAN (R 3.5.0)                  
#>  desc          1.2.0      2018-05-01 [1] CRAN (R 3.5.0)                  
#>  devtools      2.0.1      2018-10-26 [1] CRAN (R 3.5.1)                  
#>  digest        0.6.18     2018-10-10 [1] CRAN (R 3.5.1)                  
#>  dplyr         0.7.8      2018-11-10 [1] CRAN (R 3.5.1)                  
#>  drake       * 6.1.0.9000 2018-11-30 [1] Github (ropensci/drake@cb5d175) 
#>  evaluate      0.12       2018-10-09 [1] CRAN (R 3.5.1)                  
#>  formatR       1.5        2017-04-25 [1] CRAN (R 3.5.0)                  
#>  fs            1.2.6      2018-08-23 [1] CRAN (R 3.5.1)                  
#>  glue          1.3.0      2018-07-17 [1] CRAN (R 3.5.1)                  
#>  htmltools     0.3.6      2017-04-28 [1] CRAN (R 3.5.0)                  
#>  igraph        1.2.2      2018-07-27 [1] CRAN (R 3.5.1)                  
#>  knitr         1.20       2018-02-20 [1] CRAN (R 3.5.0)                  
#>  magrittr      1.5        2014-11-22 [1] CRAN (R 3.5.0)                  
#>  MASS          7.3-51.1   2018-11-01 [1] CRAN (R 3.5.1)                  
#>  memoise       1.1.0      2017-04-21 [1] CRAN (R 3.5.0)                  
#>  pillar        1.3.0      2018-07-14 [1] CRAN (R 3.5.1)                  
#>  pkgbuild      1.0.2      2018-10-16 [1] CRAN (R 3.5.1)                  
#>  pkgconfig     2.0.2      2018-08-16 [1] CRAN (R 3.5.1)                  
#>  pkgload       1.0.2      2018-10-29 [1] CRAN (R 3.5.1)                  
#>  prettyunits   1.0.2      2015-07-13 [1] CRAN (R 3.5.0)                  
#>  processx      3.2.0      2018-08-16 [1] CRAN (R 3.5.1)                  
#>  ps            1.2.1      2018-11-06 [1] CRAN (R 3.5.1)                  
#>  purrr         0.2.5      2018-05-29 [1] CRAN (R 3.5.0)                  
#>  R6            2.3.0      2018-10-04 [1] CRAN (R 3.5.1)                  
#>  Rcpp          1.0.0      2018-11-07 [1] CRAN (R 3.5.1)                  
#>  remotes       2.0.2      2018-10-30 [1] CRAN (R 3.5.1)                  
#>  rlang         0.3.0.1    2018-10-25 [1] CRAN (R 3.5.1)                  
#>  rmarkdown     1.10       2018-06-11 [1] CRAN (R 3.5.0)                  
#>  rprojroot     1.3-2      2018-01-03 [1] CRAN (R 3.5.0)                  
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.5.1)                  
#>  storr         1.2.1      2018-10-18 [1] CRAN (R 3.5.1)                  
#>  stringi       1.2.4      2018-07-20 [1] CRAN (R 3.5.0)                  
#>  stringr       1.3.1      2018-05-10 [1] CRAN (R 3.5.0)                  
#>  testthat      2.0.1      2018-10-13 [1] CRAN (R 3.5.1)                  
#>  tibble        1.4.2      2018-01-22 [1] CRAN (R 3.5.0)                  
#>  tidyselect    0.2.5      2018-10-11 [1] CRAN (R 3.5.1)                  
#>  usethis       1.4.0      2018-08-14 [1] CRAN (R 3.5.1)                  
#>  withr         2.1.2.9000 2018-11-19 [1] Github (jimhester/withr@be57595)
#>  yaml          2.2.0      2018-07-25 [1] CRAN (R 3.5.1)                  
#> 
#> [1] C:/Users/grenie/Documents/R/R-3.5.1/library

[wlandau]

This is an interesting edge case. Apparently, we need dudi1 to compute omi.

library(drake)
library(ade4)
data(doubs)
dudi1 <- dudi.pca(doubs$env, scale = TRUE, scan = FALSE, nf = 3)
nic1 <- niche(dudi1, doubs$fish, scann = FALSE)
omi <- niche.param(nic1)
rm(dudi1)
omi <- niche.param(nic1)
#> Error in eval(expr, p): object 'dudi1' not found

^{Created on 2018-11-30 by the reprex package (v0.2.1)}

This is because niche.param() does some metaprogramming.

niche.param
#> function (x) 
#> {
#>     if (!inherits(x, "niche")) 
#>         stop("Object of class 'niche' expected")
#>     appel <- as.list(x$call)
#>     X <- eval.parent(appel[[2]])$tab
#>     Y <- eval.parent(appel[[3]])
#> ...

And in our case, appel[[2]] is really dudi1, so omi depends on dudi1. However, drake does not know about this dependency relationship because dudi1 is not explicitly mentioned in the command for omi or the niche.param() function.

library(drake)
library(ade4)
data(doubs)
my_plan = drake_plan(
  dudi1 = dudi.pca(doubs$env, scale = TRUE, scan = FALSE, nf = 3),
  nic1 = niche(dudi1, doubs$fish, scann = FALSE),
  omi = niche.param(nic1)
)
config <- drake_config(my_plan)
vis_drake_graph(config)

^{Created on 2018-11-30 by the reprex package (v0.2.1)}

Why do we fail to notice dudi1? Because drake does static code analysis to detect dependencies. In other words, drake detects dependency relationships by analyzing your commands and functions without actually running them. So it cannot figure out that omi depends on dudi through appel inside niche.param(). Unfortunately, this is almost certainly not going to change in future releases.

A workaround is to explicitly mention dudi1 in the command for omi.

library(drake)
library(ade4)
data(doubs)
my_plan = drake_plan(
  dudi1 = dudi.pca(doubs$env, scale = TRUE, scan = FALSE, nf = 3),
  nic1 = niche(dudi1, doubs$fish, scann = FALSE)
)
make(my_plan)
#> target dudi1
#> target nic1
other_plan = drake_plan(
  omi = {
    dudi1
    niche.param(nic1)
  }
)
plan <- rbind(my_plan, other_plan)
make(plan)
#> target omi
config <- drake_config(plan)
vis_drake_graph(config)

^{Created on 2018-11-30 by the reprex package (v0.2.1)}

In one sense, the bigger problem is actually the ade4 package. I do not believe it should try to reload symbols in niche.param() that are not necessarily still in the environment.

[Rekyt]

Woah! Thank you so much @wlandau for the explanation.
I was expecting something like that but as I am not expert on the functioning of drake I didn't know if it was fixable.

I don't think either it would be a good idea for drake to load all symbols and clearly the behavior of niche.param() is very strange.

Thanks for the solution!