Error in loadNamespace(name) :

[csgillespie]

I'm trying to profile some relatively simple code. The code below worked a few months ago:

devtools::install_github("csgillespie/efficient", args="--with-keep.source", force=TRUE)
profvis::profvis(efficient::simulate_monopoly(100))
#Error in loadNamespace(name) : 
 # there is no package called 'csgillespie-efficient-8100d26'
efficient::simulate_monopoly(100)
# [1] 1 2 1 2 3 2 1 1 3 3 7 2 4 1 2 3 4 2 4 1 4 2 2 4 4 4 4 3 2 2 6 6 2 1 0 3 0 4 2 4

A couple of other points:

• If I install efficient* via R CMD INSTALL profvis is happy.
• I've tried deleting/re-installing profvis and efficient
• I've tried this on two computers
• Command line R and RStudio
• profvis works on a standard function calls.
• Rather bizarrely, this indicates that it can find the function:
  R> profvis::profvis(efficient::simulate_monopoly(10)) Error in parse_rprof(prof_output, expr_source) : No parsing data available. Maybe your function was too fast?
  The number 10 affects the length of the simulation.

Neither of the functions are byte-compiled.

R> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.4 LTS

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8       
 [4] LC_COLLATE=en_GB.UTF-8     LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8   
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] htmlwidgets_0.7 efficient_0.1.1 magrittr_1.5    profvis_0.3.2   htmltools_0.3.5
 [6] tools_3.3.1     Rcpp_0.12.7     stringi_1.1.1   stringr_1.1.0   digest_0.6.10  

---

A bit more info:

> profvis(simulate_monopoly(10000))
 Hide Traceback

 Rerun with Debug
 Error in loadNamespace(name) : 
  there is no package called 'csgillespie-efficient-8100d26' 
14.stop(e) 
13.value[[3L]](cond) 
12.tryCatchOne(expr, names, parentenv, handlers[[1L]]) 
11.tryCatchList(expr, classes, parentenv, handlers) 
10.tryCatch(loadNamespace(name), error = function(e) stop(e)) 
9.getNamespace(ns) 
8.asNamespace(pkg) 
7.load_pkg_into_cache(pkg, srcref_cache) 
6.extract_source_from_srcref(filename, srcref_cache) 
5.FUN(X[[i]], ...) 
4.lapply(filenames, function(filename) {
    if (filename == "<expr>") {
        return(expr_source)
    } ... 
3.get_file_contents(filenames, expr_source) 
2.parse_rprof(prof_output, expr_source) 
1.profvis(simulate_monopoly(10000)) 

[wch]

Hm, I think it's because the directory name is different from what profvis is looking for.

@hadley was there a change in devtools so that packages are built in a directory like /tmp/Rtmpj6OtK6/devtools291e347e813d/csgillespie-efficient-8100d26/ instead of /tmp/Rtmpj6OtK6/devtools291e347e813d/efficient/?

> options(error=recover)
> profvis::profvis(efficient::simulate_monopoly(100))
Error in loadNamespace(name) : 
  there is no package called ‘csgillespie-efficient-8100d26’

Enter a frame number, or 0 to exit   

 1: profvis::profvis(efficient::simulate_monopoly(100))
 2: parse_rprof(prof_output, expr_source)
 3: get_file_contents(filenames, expr_source)
 4: lapply(filenames, function(filename) {
    if (filename == "<expr>") {
        retu
 5: FUN(X[[i]], ...)
 6: extract_source_from_srcref(filename, srcref_cache)
 7: load_pkg_into_cache(pkg, srcref_cache)
 8: asNamespace(pkg)
 9: getNamespace(ns)
10: tryCatch(loadNamespace(name), error = function(e) stop(e))
11: tryCatchList(expr, classes, parentenv, handlers)
12: tryCatchOne(expr, names, parentenv, handlers[[1]])
13: value[[3]](cond)

Selection: 
Enter an item from the menu, or 0 to exit
Selection: 6
Called from: eval(substitute(browser(skipCalls = skip), list(skip = 7 - which)), 
    envir = sys.frame(which))
Browse[1]> pkg
[1] "csgillespie-efficient-8100d26"
Browse[1]> filename
[1] "/tmp/Rtmpj6OtK6/devtools291e347e813d/csgillespie-efficient-8100d26/R/monopoly.R"

[hadley]

@jimhester is more likely to know than me

[wch]

This is the content of the Rprof output file:

memory profiling: GC profiling: line profiling: sample.interval=10000
#File 1: /tmp/Rtmpj6OtK6/devtools291e347e813d/csgillespie-efficient-8100d26/R/monopoly.R
:412650:763970:38287200:2783:"$.data.frame" 1#10 "$" 1#10 "move_square" 1#92 "efficient::simulate_monopoly" "force" "doTryCatch" "tryCatchOne" "tryCatchList" "doTryCatch" "tryCatchOne" "tryCatchList" "tryCatch" "profvis::profvis" 
:427739:775998:41458144:2130:"as.list" "as.matrix.data.frame" "as.matrix" "apply" 1#7 "move_square" 1#92 "efficient::simulate_monopoly" "force" "doTryCatch" "tryCatchOne" "tryCatchList" "doTryCatch" "tryCatchOne" "tryCatchList" "tryCatch" "profvis::profvis" 
:453275:796331:46867128:3615:1#91 "efficient::simulate_monopoly" "force" "doTryCatch" "tryCatchOne" "tryCatchList" "doTryCatch" "tryCatchOne" "tryCatchList" "tryCatch" "profvis::profvis" 
:471064:810630:50589280:2502:"aperm.default" "aperm" "apply" 1#7 "move_square" 1#92 "efficient::simulate_monopoly" "force" "doTryCatch" "tryCatchOne" "tryCatchList" "doTryCatch" "tryCatchOne" "tryCatchList" "tryCatch" "profvis::profvis" 

The problem is that profvis tries to infer the package name from the path, but in this case the directory name isn't the same as the package name. (We can't always count on the user calling pkgname::function() as was done in this particular case.)

[wch]

It looks like the zip files from GitHub use that convention (like csgillespie-efficient-8100d26/) for directory naming now. Profvis could add some heuristics for inferring the package name, but unfortunately it won't be 100% reliable.

[csgillespie]

When you say package name, I presume you mean efficient and not csgillespie-efficient-8100d26. If so, what about adding an optional argument to profvis, say package. If you can't detect the package, then just print a message to the screen asking for more info.

[wch]

I'd also prefer not to require people to have to manually enter a package, and also there could be multiple packages installed that way. Also, if it required an option, I'm not sure how that would be exposed in the RStudio IDE for people who want to start profiling via the GUI.

When I say not 100% reliable, I mean that it will probably work for 99% of cases. What I have in mind won't work if the repo name differs from the package name, and I don't know if it will work for packages from other sources (like bitbucket).

[kevinushey]

I'm also bumping into this issue. Note that this occurs for me with the current devtools CRAN release.

EDIT: I think I'm bumping into something slightly different so I'll record that in a separate issue.

[kevinushey]

Note that detecting the package name based on the repository name won't work in general (e.g. the package roxygen2 is provided from the klutometis/roxygen repository). I think the only way this can be done in general is to find and parse the associated DESCRIPTION file.

[wch]

@kevinushey that's one of the of cases in the 1% that I had in mind. :-/

Here's the challenge: given the data in the Rprof data file, how do you find the DESCRIPTION file? This is what we have:

/tmp/Rtmpj6OtK6/devtools291e347e813d/csgillespie-efficient-8100d26/R/monopoly.R

If the directory were something like klutometis-roxygen-0123abc, then how do you find the DESCRIPTION file? Note that the temp directory (probably) no longer exists when profvis() is run. This is the path to the file when the package was built and installed locally.

An alternative is to have devtools rename the directory to match what's in the DESCRIPTION before building and installing.

[kevinushey]

Ack, good point! If that's the case then perhaps a failed attempt to load the namespace of an inferred package should not be an error? (I think it would just imply we don't have source references; is that correct?)

It would also be nice if devtools could help out here; it seems like there's not much that could be done on the profvis side.

[jimhester]

The issue is these RProf is returning the name of the source file alias, not the original source file. You can get the latter from the passed function directly, which will also get you the package name.

devtools::install_github("csgillespie/efficient", args="--with-keep.source", force=TRUE)
library(efficient)
attr(attr(simulate_monopoly, "srcref"), "srcfile")$original
#> /Users/jhester/Library/R/3.3/library/efficient/R/efficient

[jimhester]

I should note this isn't really a devtools issue, you will see the same behavior anytime you install a package from a directory with a different name than the package name. We could change the devtools behavior, but this case would still remain.

git2r::clone("https://github.com/csgillespie/efficient", "test")
install.packages("test", repos = NULL, type = "source", INSTALL_opts = "--with-keep.source")
library(efficient)
attr(attr(simulate_monopoly, "srcref"), "srcfile")
#> /.../R/test/R/monopoly.R

[wch]

@jimhester Do you see a way to get the package name we don't have the actual function object? That could happen if you ran profvis(fun()) and fun() in turn called efficient::simulate_monopoly.

[wch]

I've added some heuristics that will work where the package directory is xxx-pkgname-yyy. It won't work for cases like roxygen2, where the package name differs from the dir name (like xxx-roxygen-yyy), nor will it work for the test case that @jimhester raised. But this should cover the vast majority of cases.

[jimhester]

@wch You can get a mapping of all files used in loaded namespaces along with their package name with the following, which can then be used to find the figure out what namespace to load. It is basically instantaneous to run.

# helper to extract captures
regcaptures <- function(x, m) {
  ind <- !is.na(m) & m > -1L
  so <- attr(m, "capture.start")[ind]
  eo <- so + attr(m, "capture.length")[ind] - 1L
  substring(x[ind], so ,eo)
}

package_map <- function(loaded_ns = loadedNamespaces()) {
  res <- lapply(loaded_ns, function(ns) {

    # Get the first exported function, need to filter out S4 mangled names however (.__T__)
    first_export <- head(grep("^\\.__[TC]__", getNamespaceExports(ns), value = TRUE, invert = TRUE, fixed = FALSE), n = 1)
    if (length(first_export) > 0) {
      f <- get(first_export, envir = asNamespace(ns), inherits = FALSE)
      src <- getSrcref(f)
      if (!is.null(src)) {

        # retrieve all of the source lines from the installed namespace, which
        # have the source line directives included
        lines <- getSrcLines(attr(src, "srcfile"), 1, Inf)

        # Find and extract the line directives
        m <- regexpr("^#line \\d+ \"(.*)\"$", lines, perl = TRUE)
        regcaptures(lines, m)
  }}})
  data.frame(filename = unlist(res), package = rep(loaded_ns, times = lengths(res)))
}
map <- package_map()
head(map)
#>   filename
#> 1 /.../test/R/datasets.R                                                     package
#> 2 /.../test/R/monopoly.R                                                   efficient
#> 3 /.../test/R/snakes_and_ladders.R                                         efficient
#> 4 /.../test/R/test_rcpp.R                                                  efficient
#> 5 /.../RtmpOolYvL/devtools64c82ab8169f/rstats-db-DBI-73e0945/R/DBObject.R        DBI
#> 6 /.../RtmpOolYvL/devtools64c82ab8169f/rstats-db-DBI-73e0945/R/DBDriver.R        DBI

[wch]

@jimhester That's great! It's better than the fix that I put in. I think my fix will still be useful as a fallback, in case the package in question isn't loaded. (This can happen if the Rprof file was generated outside of profvis.)