Enable caching of sourceCpp compilations across R sessions #504

jjallaire · 2016-07-11T17:42:06Z

This PR adds a cacheDir argument to sourceCpp to enable caching of shared libraries across R sessions. The argument defaults to tempdir() which emulates the existing default behavior of only caching compilations within the current R session.

The implementation changes the storage of the data structure that tracks previous compilations from a static C++ object to disk based storage (a serialized R list) within the specified cacheDir.

This feature is motivated by the desire to cache Rcpp chunks within knitr documents (particularly important for book authoring). See rstudio/bookdown#113 for additional discussion.

eddelbuettel · 2016-07-11T17:49:54Z

Regarding the Travis timeout, I am wrapping the test call in travis_wait which "works", but is annoying as all output is suppressed til done.

Do we know whether Travis is just slow? We used to get done in about 8 minutes ... so can we be sure it is not the changeset here?

coatless · 2016-07-11T17:57:53Z

Apologies in advance for not contributing anything of substance next, but...

This is awesome!

jjallaire · 2016-07-11T18:11:58Z

I think that it's the changeset. Here's why: the cache of compiled R
functions is now persisted to disk rather than stored in memory. Further,
when we lookup a function in the cache we read the entire "database" (an R
list) and then re-write the entire database after the compilation. This
introduces some O(N) behavior, which mostly doesn't matter for order of a
couple dozen C++ compilation units, but our unit tests introduce so many
C++ compilation units that we're hitting this in a material way.

Let me think about some possible ways out of this box....

On Mon, Jul 11, 2016 at 1:49 PM, Dirk Eddelbuettel <notifications@github.com

wrote:

Regarding the Travis timeout, I am wrapping the test call in travis-wait
https://github.com/eddelbuettel/rquantlib/blob/master/.travis.yml#L30-L31
which "works", but is annoying as all output is suppressed til done.

Do we know whether Travis is just slow? We used to get done in about 8
minutes ... so can we be sure it is not the changeset here?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#504 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAGXx8RhaRxbIMgDC6OgGXQc0Brxzfgfks5qUoJCgaJpZM4JJmwK
.

eddelbuettel · 2016-07-11T18:20:26Z

Good analysis. In which case we can surely just prefix with travis_wait which is documented here.

At least until we have a better fix.

jjallaire · 2016-07-11T19:43:03Z

Kevin just helped me get to the bottom of things (it was actually a bug
that was preventing 2nd and subsequent calls to sourceCpp from working
correctly, which led to failed tests and ultimately a crash).

The latest commit should get us back to green on Travis.

On Mon, Jul 11, 2016 at 2:11 PM, JJ Allaire jj.allaire@gmail.com wrote:

I think that it's the changeset. Here's why: the cache of compiled R
functions is now persisted to disk rather than stored in memory. Further,
when we lookup a function in the cache we read the entire "database" (an R
list) and then re-write the entire database after the compilation. This
introduces some O(N) behavior, which mostly doesn't matter for order of a
couple dozen C++ compilation units, but our unit tests introduce so many
C++ compilation units that we're hitting this in a material way.

Let me think about some possible ways out of this box....

On Mon, Jul 11, 2016 at 1:49 PM, Dirk Eddelbuettel <
notifications@github.com> wrote:

Regarding the Travis timeout, I am wrapping the test call in travis-wait
https://github.com/eddelbuettel/rquantlib/blob/master/.travis.yml#L30-L31
which "works", but is annoying as all output is suppressed til done.

Do we know whether Travis is just slow? We used to get done in about 8
minutes ... so can we be sure it is not the changeset here?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#504 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAGXx8RhaRxbIMgDC6OgGXQc0Brxzfgfks5qUoJCgaJpZM4JJmwK
.

eddelbuettel · 2016-07-11T19:45:53Z

Yay yay yay -- I was starting to fear you two would get turned off working on this forever and ever :)

eddelbuettel · 2016-07-12T12:30:20Z

I just merged #503 which you can probably bring in here too.

eddelbuettel · 2016-07-12T12:39:28Z

Running a rev.dep now with this PR #504 and the just merged #503.

…er)" This reverts commit f04c9ed.

jjallaire · 2016-07-12T17:57:58Z

Note that you'll need to update from the Rcpp branch again to try this out.

eddelbuettel · 2016-07-12T18:13:01Z

Nice work -- using the Fibonacci example referenced by @yihui:

R> library(Rcpp)
R> sourceCpp("fib.cpp", cacheDir=".")
R> sourceCpp("fib.cpp", cacheDir=".")
R> sourceCpp("fib.cpp", cacheDir=".")
R> system("tree")
.
├── fib.cpp
└── sourceCpp-0.12.5.6-x86_64-pc-linux-gnu-70800
    ├── cache.rds
    ├── sourcecpp_209b23f927e0
    │   ├── fib.cpp
    │   ├── fib.cpp.R
    │   ├── fib.o
    │   └── sourceCpp_2.so
    └── token.rds

2 directories, 7 files
R>

yihui · 2016-07-12T18:22:55Z

I quickly inspected cache.rds and fib.cpp.R, and saw absolute paths were used. I wonder if it is okay to just use relative paths, because absolute paths means you cannot move the cache directory to other places.

jjallaire · 2016-07-12T18:44:31Z

I agree that it would be better to use relative paths here however the
absolute path assumption runs pretty deep through the sourceCpp
implementation so I think it would be too risky to try to unwind it right
now.

On Tue, Jul 12, 2016 at 2:22 PM, Yihui Xie notifications@github.com wrote:

I quickly inspected cache.rds and fib.cpp.R, and saw absolute paths were
used. I wonder if it is okay to just use relative paths, because absolute
paths means you cannot move the cache directory to other places.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#504 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAGXx_mFtO6NNvlIM3dU1OUTg-f0QQZJks5qU9t_gaJpZM4JJmwK
.

dcdillon · 2016-07-12T20:06:03Z

What happens if you sourceCpp on two files with the same relative name in the same session? After a setwd for instance.

jjallaire · 2016-07-12T20:49:24Z

That will work fine (the cache is keyed by absolute path not relative path).

jjallaire · 2016-07-12T20:51:55Z

Note that this PR doesn't change anything about how the cache works other than it's now stored on disk rather than in memory. For cacheDir = tempdir() (the default) this will result in identical behavior to the current system. For cacheDir = ~/tmp/mycache it will result in a cache that persists across sessions. The motivation for this is interoperability with the knitr cache (see yihui/knitr#1239)

kevinushey · 2016-07-12T21:12:37Z

What about allowing users to set a cache directory with an R option or something like that? E.g. rcpp.cache.dir?

eddelbuettel · 2016-07-12T21:15:45Z

I like that idea.

All it would take is to change

cacheDir = tempDir()

to

cacheDir = getOption("rcpp.cache.dir"), tempDir() )

The second argument (default=NULL) for getOption() is really useful.

jjallaire · 2016-07-12T21:38:15Z

Okay, just added this.

On Tue, Jul 12, 2016 at 5:15 PM, Dirk Eddelbuettel <notifications@github.com

wrote:

I like that idea.

All it would take is to change

cacheDir = tempDir()

to

cacheDir = getOption("rcpp.cache.dir"), tempDir()

The second argument (default=NULL) for getOption() is really useful.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#504 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAGXxw_bAil3wGSR3cjqZPPIihH57Yvvks5qVAQBgaJpZM4JJmwK
.

eddelbuettel · 2016-07-13T03:29:22Z

This too should be ready per the reverse depends check summarized here so I will merge this.

yihui · 2016-07-13T08:52:08Z

Cool. The knitr PR was also merged, so ````{Rcpp cache=TRUE}` is ready to be used with the dev versions of knitr and Rcpp now.

jjallaire added 6 commits July 11, 2016 11:45

initial work on cacheDir

f3d4961

fix typo in path list element

4d4f1a1

use double for lastModified rather than time_t

669d72b

add cacheDir to usage in Rd files

4bd5104

get cache varible w/o direct reference (R CMD check)

566832a

use cacheDir for code temporary file

2fc52c5

jjallaire mentioned this pull request Jul 11, 2016

Enable caching of Rcpp engine chunks yihui/knitr#1239

Merged

jjallaire added 2 commits July 11, 2016 14:55

don't compress dynlib cache

f7e1024

test for nzchar rather than !is.null

7419dfc

jjallaire added 2 commits July 12, 2016 08:18

return buildDirectory with sourceCpp results

a882203

don't use cacheDir for code temp file (creates permanent litter)

f04c9ed

jjallaire added 4 commits July 12, 2016 12:00

correctly insert code into cache

6f43bb5

Revert "don't use cacheDir for code temp file (creates permanent litt…

dc17457

…er)" This reverts commit f04c9ed.

check that cppSourcePath exists before returning cache entry

cea0372

bump version

85fc4c2

add cleanupCacheDir option to sourceCpp

a825e48

jjallaire added 2 commits July 12, 2016 17:35

bump version

f22dddb

add rcpp.cache.dir option for global cacheDir setting

a06980d

eddelbuettel merged commit cf4c162 into master Jul 13, 2016

Uh oh!

Enable caching of sourceCpp compilations across R sessions #504

Enable caching of sourceCpp compilations across R sessions #504

Uh oh!

Conversation

jjallaire commented Jul 11, 2016

Uh oh!

eddelbuettel commented Jul 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coatless commented Jul 11, 2016

Uh oh!

jjallaire commented Jul 11, 2016

Uh oh!

eddelbuettel commented Jul 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjallaire commented Jul 11, 2016

Uh oh!

eddelbuettel commented Jul 11, 2016

Uh oh!

eddelbuettel commented Jul 12, 2016

Uh oh!

eddelbuettel commented Jul 12, 2016

Uh oh!

jjallaire commented Jul 12, 2016

Uh oh!

eddelbuettel commented Jul 12, 2016

Uh oh!

yihui commented Jul 12, 2016

Uh oh!

jjallaire commented Jul 12, 2016

Uh oh!

dcdillon commented Jul 12, 2016

Uh oh!

jjallaire commented Jul 12, 2016

Uh oh!

jjallaire commented Jul 12, 2016

Uh oh!

kevinushey commented Jul 12, 2016

Uh oh!

eddelbuettel commented Jul 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjallaire commented Jul 12, 2016

Uh oh!

eddelbuettel commented Jul 13, 2016

Uh oh!

yihui commented Jul 13, 2016

Uh oh!

Uh oh!

eddelbuettel commented Jul 11, 2016 •

edited

Loading

eddelbuettel commented Jul 11, 2016 •

edited

Loading

eddelbuettel commented Jul 12, 2016 •

edited

Loading