.future folder

[marcosci]

Hi Henrik,

do you think it would be possible/worth having auxiliary functions giving more control over the .future folder? The problem in my case was/is that I only have a certain quota on our HPC and if .futures is located in my personal drive and I work with spatial data - I reach this quota quite fast.

I was not able to figure out how to control this with future.batchtools directly and just relied on .Rprofile to set my working directory to /scratch while loading R. This ensured that the default call of makeRegistry from batchtools chooses this directory for the .future folder. However, a more reproducible workflow for that would be nice.

Or did I just miss a way to do this with future.batchtools?

Cheers
Marco

[HenrikBengtsson]

Could you use a symbolic file link, e.g.

$ cd /path/to/project
$ mkdir /scratch/.future
$ ln -s /scratch/.future .future
$ R

or from within R:

$ cd /path/to/project
$ R
> dir.create("/scratch/.future")
> file.symlink(from = "/scratch/.future", to = ".future")
> 

[marcosci]

Just tried it, it still creates the folders for the jobs in /home, rather than /scratch.

[HenrikBengtsson]

Please try to give more details on exactly what you've done. I don't understand why it should create a folder elsewhere (other than the current directory). Also, if one already exists in the current directory (even a symbol link) it will not remove it and create another one.

Reproducible example on Linux:

$ mkdir ~/my_project
$ cd ~/my_project
$ mkdir /tmp/.future   # Target location
$ ln -s /tmp/.future .future   # Local directory location
ls -la
total 20
drwxr-xr-x  2 hb hb  4096 Aug 31 08:46 .
drwxrwxr-x 54 hb hb 12288 Aug 31 08:36 ..
lrwxrwxrwx  1 hb hb    12 Aug 31 08:36 .future -> /tmp/.future

$ R
[...]
> library(future.batchtools)
> plan(batchtools_local)
> dir(".future")
character(0)
> dir("/tmp/.future")
character(0)
> f <- future(42)
> dir(".future")
[1] "20180831_084504-dz0Xfl"
> dir("/tmp/.future")
[1] "20180831_084504-dz0Xfl"
> value(f)
[1] 42
> quit("no")

$ pwd
/home/hb/my_project
$ ls -la
total 20
drwxr-xr-x  2 hb hb  4096 Aug 31 08:46 .
drwxrwxr-x 54 hb hb 12288 Aug 31 08:36 ..
lrwxrwxrwx  1 hb hb    12 Aug 31 08:36 .future -> /tmp/.future

[HenrikBengtsson]

FYI, you can control the location of the .future folder via environment variable R_FUTURE_CACHE_PATH, e.g. add

R_FUTURE_CACHE_PATH=/path/to/cache/.future

to your ~/.Renviron file.

[HenrikBengtsson]

Updates in the develop branch: The relative or absolute location of the .future folder can now be configured using option future.cache.path or environment variable R_FUTURE_CACHE_PATH. This is now documented in:

help("future.batchtools.options", package = "future.batchtools")

[coatless]

Woohoo! Finally landed on the right ticket. @HenrikBengtsson maybe add a link to the environment variable configurations in the README? Sadly, R_FUTURE_CACHE_PATH was near impossible to find from a search.

To increase visibility of this ticket from a search engine:

• future r programming change .future directory location
• modify location of .future/ directory away from $HOME/.future
• change location of .future directory on cluster

[HenrikBengtsson]

That's a fair issue-tracker hack :)

FWIW, future.cache.path and R_FUTURE_CACHE_PATH are documented in ?future.batchtools::future.batchtools.options but obviously it's hard to find them unless you know you should search for them.

I'll see what I can do for documention.