Working directory set with withr::with_dir() not respected in multisession

[lorenzwalthert]

With current CRAN releases if future and future.apply:

##  ............................................................................
##  the working directory is not set to a temp dir with withr::with_dir()   ####

get_wd_from_temp_dir <- function() {
  withr::with_dir(tempdir(), {
    future.apply::future_sapply(1:2, function(x) {
      print(getwd())
    })
  })
}

future::plan(future::sequential)
get_wd_from_temp_dir() # works
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [2] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
future::plan(future::multisession)
getwd()
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/Rtmprio1ap/reprexbb0b1f8de0d9"
get_wd_from_temp_dir() # does not work
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/Rtmprio1ap/reprexbb0b1f8de0d9"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/Rtmprio1ap/reprexbb0b1f8de0d9"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/Rtmprio1ap/reprexbb0b1f8de0d9"
#> [2] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/Rtmprio1ap/reprexbb0b1f8de0d9"

##  ............................................................................
##  Everything seems to work when setting it the conventional way           ####


get_wd_outside_temp_dir <- function() {
  future.apply::future_sapply(1:2, function(x) {
    print(getwd())
  })
}
old <- setwd(tempdir())
future::plan(future::sequential)
get_wd_outside_temp_dir()
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [2] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
future::plan(future::multisession)
getwd()
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"

get_wd_outside_temp_dir() # also works
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [1] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"
#> [2] "/private/var/folders/6g/bg5mcmsj7s51r33snbxhm1400000gn/T/RtmpDFg23u"

^{Created on 2020-03-05 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.1 (2019-07-05)
#>  os       macOS Catalina 10.15.3      
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Europe/Zurich               
#>  date     2020-03-05                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version date       lib source        
#>  assertthat     0.2.1   2019-03-21 [1] CRAN (R 3.6.0)
#>  backports      1.1.5   2019-10-02 [1] CRAN (R 3.6.0)
#>  callr          3.4.2   2020-02-12 [1] CRAN (R 3.6.0)
#>  cli            2.0.1   2020-01-08 [1] CRAN (R 3.6.0)
#>  codetools      0.2-16  2018-12-24 [1] CRAN (R 3.6.1)
#>  crayon         1.3.4   2017-09-16 [1] CRAN (R 3.6.0)
#>  desc           1.2.0   2018-05-01 [1] CRAN (R 3.6.0)
#>  devtools       2.2.2   2020-02-17 [1] CRAN (R 3.6.0)
#>  digest         0.6.24  2020-02-12 [1] CRAN (R 3.6.0)
#>  ellipsis       0.3.0   2019-09-20 [1] CRAN (R 3.6.0)
#>  evaluate       0.14    2019-05-28 [1] CRAN (R 3.6.0)
#>  fansi          0.4.1   2020-01-08 [1] CRAN (R 3.6.0)
#>  fs             1.3.1   2019-05-06 [1] CRAN (R 3.6.0)
#>  future         1.16.0  2020-01-16 [1] CRAN (R 3.6.0)
#>  future.apply   1.4.0   2020-01-07 [1] CRAN (R 3.6.0)
#>  globals        0.12.5  2019-12-07 [1] CRAN (R 3.6.0)
#>  glue           1.3.1   2019-03-12 [1] CRAN (R 3.6.0)
#>  highr          0.8     2019-03-20 [1] CRAN (R 3.6.0)
#>  htmltools      0.4.0   2019-10-04 [1] CRAN (R 3.6.0)
#>  knitr          1.28    2020-02-06 [1] CRAN (R 3.6.0)
#>  listenv        0.8.0   2019-12-05 [1] CRAN (R 3.6.0)
#>  magrittr       1.5     2014-11-22 [1] CRAN (R 3.6.0)
#>  memoise        1.1.0   2017-04-21 [1] CRAN (R 3.6.0)
#>  pkgbuild       1.0.6   2019-10-09 [1] CRAN (R 3.6.0)
#>  pkgload        1.0.2   2018-10-29 [1] CRAN (R 3.6.0)
#>  prettyunits    1.1.1   2020-01-24 [1] CRAN (R 3.6.1)
#>  processx       3.4.2   2020-02-09 [1] CRAN (R 3.6.1)
#>  ps             1.3.2   2020-02-13 [1] CRAN (R 3.6.0)
#>  R6             2.4.1   2019-11-12 [1] CRAN (R 3.6.0)
#>  Rcpp           1.0.3   2019-11-08 [1] CRAN (R 3.6.0)
#>  remotes        2.1.1   2020-02-15 [1] CRAN (R 3.6.0)
#>  rlang          0.4.4   2020-01-28 [1] CRAN (R 3.6.1)
#>  rmarkdown      2.1     2020-01-20 [1] CRAN (R 3.6.0)
#>  rprojroot      1.3-2   2018-01-03 [1] CRAN (R 3.6.0)
#>  sessioninfo    1.1.1   2018-11-05 [1] CRAN (R 3.6.0)
#>  stringi        1.4.6   2020-02-17 [1] CRAN (R 3.6.0)
#>  stringr        1.4.0   2019-02-10 [1] CRAN (R 3.6.0)
#>  testthat       2.3.1   2019-12-01 [1] CRAN (R 3.6.0)
#>  usethis        1.5.1   2019-07-04 [1] CRAN (R 3.6.0)
#>  withr          2.1.2   2018-03-15 [1] CRAN (R 3.6.0)
#>  xfun           0.12    2020-01-13 [1] CRAN (R 3.6.1)
#>  yaml           2.2.1   2020-02-01 [1] CRAN (R 3.6.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Source: r-lib/styler#277

[HenrikBengtsson]

(Transferred from the future.apply repos to the future repos, since this question/feedback/feature request applies to all of the future framework)

Yeah, that's actually expected. Given that futures can be evaluated anywhere (e.g. on my laptop if I gave you access) - not just on your local machine - it does not really make sense for a future to change the working directory to the working directory on the main R session. That directory might not exist and might not contain the same things. The same limitation goes for files, we cannot really make assumptions about files.

A potential solution to this is mentioned in the design roadmap issue 'DESIGN: Future API - Minimal/Core/Essential API and Extended/Optional API':

• Other resource specifications, such as only running on the local machine, on the local file system, on a given version of R, access to a certain set of files, and so on.

The gist is to introduce a way for the developer to say that a future requires the local file system to be resolved. Using your example, then you would be able to do something like:

opwd <- getwd()
y <- future.apply::future_sapply(1:2, function(x) {
    setwd(opwd)
    print(getwd())
  }, future.resources = local_folder(opwd))

This is far into the future before anything like this is available. Related to this: I think drake has taken several steps toward this model in its support for specifying file dependencies etc.

[HenrikBengtsson]

To further clarify the above example: If a plan is set with remote workers, then the future.resources specification cannot be met and it'll fallback to resolve the futures sequentially on the local machine. A fancier version of this would be to fallback to a parallel plan on the local machine - that would require an extension that allows us to specify multiple alternative plans with different properties/resources and priorities.

[lorenzwalthert]

Thanks a lot for the detailed clarification @HenrikBengtsson! It all makes sense now. I wonder if there should be any reference to this in the package documentation or if it is too much of an exotic case. You can close the issue depending on this.