Renv shouldn't connect to repositories on startup

[kamilzyla]

Problem

It seems that on startup renv connects to all the repositories listed in the lockfile to check their status / available packages. This makes some sense when working interactively in a single R session, but becomes troublesome when you frequently launch new R sessions. The problem is especially serious, if you have multiple repositories listed in the lockfile and they happen to be unavailable - renv will incur a long delay with multiple retries for each repository.

This problem arose in a real use case. We use the box package to modularize a large Shiny app. Due to module caching in box, we need to run the app in a new R session each time we want to reload it. The app uses private packages available from a set of RSPM repositories, which can only be accessed via VPN. Sometimes, due to network issues or a configuration problem, the repositories cannot be accessed.

Proposed solution

I think renv should only connect to the repositories when it is needed, i.e. when instructed to install/update packages. This is how popular tools for creating isolated environments work in other languages, e.g. yarn in Node.js and venv in Python.

Notes

The problem could be alleviated by setting renv.config.connect.timeout to a low value, but a bug prevents it from working: this comment talks about using a shorter delay, but the code actually uses a fixed value which can be bigger than the setting provided by the user.

[kevinushey]

I suspect this is happening because of the synchronization check that renv does on startup. Does setting something like:

RENV_CONFIG_SYNCHRONIZED_CHECK = FALSE

in an appropriate .Renviron help?

[kevinushey]

Here's the big stack trace I see:

Tracing renv_available_packages_impl(type, repos, quiet) on entry 
     █
  1. └─.rs.invokeHook("rstudio.sessionInit", FALSE)
  2.   ├─base::tryCatch(...)
  3.   │ └─base:::tryCatchList(expr, classes, parentenv, handlers)
  4.   │   └─base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
  5.   │     └─base:::doTryCatch(return(expr), name, parentenv, handler)
  6.   └─renv:::fun(...)
  7.     └─renv:::renv_project_synchronized_check(project, lockfile) /Users/kevinushey/r/pkg/renv/R/load.R:586:14
  8.       └─renv::snapshot(...) /Users/kevinushey/r/pkg/renv/R/project.R:246:2
  9.         └─renv:::renv_lockfile_create(project, libpaths, type, packages) /Users/kevinushey/r/pkg/renv/R/snapshot.R:137:2
 10.           ├─`%>%`(...) /Users/kevinushey/r/pkg/renv/R/lockfile.R:156:2
 11.           │ └─base::eval(call, envir = parent.frame()) /Users/kevinushey/r/pkg/renv/R/utils.R:19:2
 12.           │   └─base::eval(call, envir = parent.frame())
 13.           ├─renv:::renv_snapshot_fixup(...)
 14.           │ └─renv:::renv_snapshot_fixup_renv(records) /Users/kevinushey/r/pkg/renv/R/snapshot.R:900:2
 15.           ├─`%>%`(...) /Users/kevinushey/r/pkg/renv/R/snapshot.R:911:2
 16.           │ └─base::eval(call, envir = parent.frame()) /Users/kevinushey/r/pkg/renv/R/utils.R:19:2
 17.           │   └─base::eval(call, envir = parent.frame())
 18.           ├─renv:::renv_snapshot_filter(...)
 19.           │ └─renv:::renv_snapshot_filter_all(project, records) /Users/kevinushey/r/pkg/renv/R/snapshot.R:757:2
 20.           └─renv:::renv_snapshot_r_packages(libpaths = libpaths, project = project) /Users/kevinushey/r/pkg/renv/R/snapshot.R:803:2
 21.             └─renv:::uapply(libpaths, renv_snapshot_r_packages_impl, project = project) /Users/kevinushey/r/pkg/renv/R/snapshot.R:507:2
 22.               ├─base::unlist(lapply(x, f, ...), recursive = FALSE) /Users/kevinushey/r/pkg/renv/R/utils-map.R:65:2
 23.               └─base::lapply(x, f, ...)
 24.                 └─renv:::FUN(X[[i]], ...)
 25.                   └─base::lapply(descriptions, renv_snapshot_description) /Users/kevinushey/r/pkg/renv/R/snapshot.R:544:2
 26.                     └─renv:::FUN(X[[i]], ...)
 27.                       └─renv:::renv_snapshot_description_source(dcf) /Users/kevinushey/r/pkg/renv/R/snapshot.R:641:2
 28.                         ├─base::local(...) /Users/kevinushey/r/pkg/renv/R/snapshot.R:686:2
 29.                         │ └─base::eval.parent(substitute(eval(quote(expr), envir)))
 30.                         │   └─base::eval(expr, p)
 31.                         │     └─base::eval(expr, p)
 32.                         └─base::eval(...)
 33.                           └─base::eval(...)
 34.                             ├─renv:::catch(renv_available_packages_latest(package)) /Users/kevinushey/r/pkg/renv/R/snapshot.R:688:4
 35.                             │ ├─base::tryCatch(...) /Users/kevinushey/r/pkg/renv/R/utils.R:167:2
 36.                             │ │ └─base:::tryCatchList(expr, classes, parentenv, handlers)
 37.                             │ │   └─base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
 38.                             │ │     └─base:::doTryCatch(return(expr), name, parentenv, handler)
 39.                             │ └─base::withCallingHandlers(expr, error = renv_error_capture)
 40.                             └─renv:::renv_available_packages_latest(package)
 41.                               ├─renv:::catch(method(package, type)) /Users/kevinushey/r/pkg/renv/R/available-packages.R:325:4
 42.                               │ ├─base::tryCatch(...) /Users/kevinushey/r/pkg/renv/R/utils.R:167:2
 43.                               │ │ └─base:::tryCatchList(expr, classes, parentenv, handlers)
 44.                               │ │   └─base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
 45.                               │ │     └─base:::doTryCatch(return(expr), name, parentenv, handler)
 46.                               │ └─base::withCallingHandlers(expr, error = renv_error_capture)
 47.                               └─renv:::method(package, type)
 48.                                 └─renv:::renv_available_packages_latest_repos_impl(package, "binary") /Users/kevinushey/r/pkg/renv/R/available-packages.R:412:2
 49.                                   └─renv:::renv_available_packages(type = type, quiet = TRUE) /Users/kevinushey/r/pkg/renv/R/available-packages.R:247:2
 50.                                     ├─renv:::renv_timecache(...) /Users/kevinushey/r/pkg/renv/R/available-packages.R:18:2
 51.                                     │ └─renv:::renv_timecache_set(key, value) /Users/kevinushey/r/pkg/renv/R/timecache.R:13:4
 52.                                     │   └─renv:::renv_timecache_entry(value) /Users/kevinushey/r/pkg/renv/R/timecache.R:39:2
 53.                                     └─renv:::renv_available_packages_impl(type, repos, quiet) /Users/kevinushey/r/pkg/renv/R/timecache.R:45:2

And this is all ultimately coming from here:

renv/R/snapshot.R

Lines 676 to 689 in 9344caf

	# NOTE: this is sort of a hack that allows renv to declare packages which
	# appear to be installed from sources, but are actually available on the
	# active R package repositories, as though they were retrieved from that
	# repository. however, this is often what users intend, especially if
	# they haven't configured their repository to tag the packages it makes
	# available with the 'Repository:' field in the DESCRIPTION file.
	#
	# still, this has the awkward side-effect of a package's source potentially
	# depending on what repositories happen to be active at the time of snapshot,
	# so it'd be nice to tighten up the logic here if possible
	entry <- local({
	renv_scope_options(renv.verbose = FALSE)
	catch(renv_available_packages_latest(package))
	})

So the most likely explanation is that you have one or more packages that renv wasn't able to infer the source from locally, and so it fell back to the (rather hacky approach) of checking if there is a package of the same name available on CRAN.

[tedmoorman]

and so it fell back to the (rather hacky approach) of checking if there is a package of the same name available on CRAN.

Yikes! And I think this may be super-problematic for air-gapped systems.

[tedmoorman]

Does setting something like:

RENV_CONFIG_SYNCHRONIZED_CHECK = FALSE

in an appropriate .Renviron help?

It does seem to help, so far ...

[orderlyquant]

Just found this thread today, after experiencing startup delays for some time. Setting RENV_CONFIG_SYNCHRONIZED_CHECK = FALSE in .Renviron seems to resolve. Given the subtlety of the issue and the obscurity of the fix, I second @kamilzyla's original post.

[hadley]

I think this problem has been fixed because renv_project_synchronized_check() now calls renv_snapshot_library() instead of snapshot(). As far as I can tell, that function only uses local data:

renv/R/snapshot.R

Lines 583 to 603 in fab327a

	library <- renv_path_normalize(library %||% renv_libpaths_active())
	paths <- list.files(library, full.names = TRUE)
	# remove 'base' packages
	paths <- paths[!basename(paths) %in% renv_packages_base()]
	# remove ignored packages
	ignored <- renv_project_ignored_packages(project = project)
	paths <- paths[!basename(paths) %in% ignored]
	# remove paths that are not valid package names
	pattern <- sprintf("^%s$", .standard_regexps()$valid_package_name)
	paths <- paths[grep(pattern, basename(paths))]
	# validate the remaining set of packages
	valid <- renv_snapshot_library_diagnose(library, paths)
	# remove duplicates (so only first package entry discovered in library wins)
	duplicated <- duplicated(basename(valid))
	packages <- valid[!duplicated]

[kevinushey]

I think this can still happen if a package has an unknown source, since we call status here:

renv/R/project.R

Line 314 in 4c5a62f

status(project = project, sources = FALSE)

and we have this bit of code:

renv/R/snapshot.R

Lines 764 to 808 in 4c5a62f

	# NOTE: this is sort of a hack that allows renv to declare packages which
	# appear to be installed from sources, but are actually available on the
	# active R package repositories, as though they were retrieved from that
	# repository. however, this is often what users intend, especially if
	# they haven't configured their repository to tag the packages it makes
	# available with the 'Repository:' field in the DESCRIPTION file.
	#
	# still, this has the awkward side-effect of a package's source potentially
	# depending on what repositories happen to be active at the time of snapshot,
	# so it'd be nice to tighten up the logic here if possible
	#
	# NOTE: local sources are also searched here as part of finding the 'latest'
	# available package, so we need to handle local packages discovered here
	tryCatch(
	renv_snapshot_description_source_hack(package),
	error = function(e) list(Source = "unknown")
	)
	}
	renv_snapshot_description_source_hack <- function(package) {
	for (type in renv_package_pkgtypes()) {
	# check cellar
	cellar <- renv_available_packages_cellar(type)
	if (package %in% cellar$Package)
	return(list(Source = "Cellar"))
	# check available packages
	dbs <- available_packages(type = type, quiet = TRUE)
	for (i in seq_along(dbs)) {
	if (package %in% dbs[[i]]$Package) {
	return(list(
	Source = "Repository",
	Repository = names(dbs)[[i]]
	))
	}
	}
	}
	list(Source = "unknown")
	}

Perhaps we need a toggle to disable that code during the synchronization check?