compatibility with pkgload::load_all()

[wlandau]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

@krlmlr, we have been emailing about compatibility between targets and pkgload::load_all(). I posted this discussion to make it easier to share code.

Let's say we have a package with this DESCRIPTION file:

Package: test
Type: Package
Title: What the Package Does (Title Case)
Version: 0.1.0
Author: Who wrote it
Maintainer: The package maintainer <yourself@somewhere.net>
Description: More about what it does (maybe more than one line)
    Use four spaces when indenting paragraphs within the Description.
License: What license is it under?
Encoding: UTF-8
LazyData: true
RoxygenNote: 7.3.0

and this R/functions.R file:

f <- function() {
  Sys.getpid()
}

If we were to pursue integration with load_all(), I believe we should use a wrapper function around existing functionality for installed packages: https://books.ropensci.org/targets/packages.html#r-packages-as-projects. The wrapper I propose is tar_load_all() defined below, which sets the envir, imports, and packages options so that the pipeline uses the package namespace instead of the global environment. Here is the _targets.R file:

library(targets)
tar_load_all <- function() {
  eval(parse(text = "pkgload::load_all()"))
  name <- pkgload::pkg_name()
  packages <- setdiff(tar_option_get("packages"), name)
  envir <- pkgload::pkg_ns()
  imports <- unique(c(tar_option_get("imports"), name))
  tar_option_set(packages = packages, envir = envir, imports = imports)
  invisible()
}
tar_load_all()
print(Sys.getpid())
list(
  tar_target(x, f())
)

When I inspect and run the pipeline locally, everything seems to work.

library(targets)
tar_visnetwork() # Verify that the static code analysis detects f().

tar_make()
#> ℹ Loading test
#> [1] 36475
#> ▶ dispatched target x
#> ● completed target x [0 seconds]
#> ▶ completed pipeline [0.051 seconds]

tar_read(x)
#> [1] 36475

However, when the target runs in a crew worker, the functions in the namespace are missing.

# _targets.R file with a {crew} controller:
library(targets)
tar_load_all <- function() {
  eval(parse(text = "pkgload::load_all()"))
  name <- pkgload::pkg_name()
  packages <- setdiff(tar_option_get("packages"), name)
  envir <- pkgload::pkg_ns()
  imports <- unique(c(tar_option_get("imports"), name))
  tar_option_set(packages = packages, envir = envir, imports = imports)
  invisible()
}
tar_load_all()

# Add a {crew} controller for parallel computing:
tar_option_set(controller = crew::crew_controller_local(workers = 1L))

print(Sys.getpid())
list(
  tar_target(x, f())
)

# R console:
tar_destroy()
tar_make()
#> ℹ Loading test
#> [1] 36576
#> ▶ dispatched target x
#> ✖ errored target x
#> ✖ errored pipeline [1.638 seconds]
#> Error:
#> ! Error running targets::tar_make()
#> Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
#> Debugging guide: https://books.ropensci.org/targets/debugging.html
#> How to ask for help: https://books.ropensci.org/targets/help.html
#> Last error message:
#>     could not find function "f"
#> Last error traceback:
#>     .handleSimpleError(function (condition)  {     state$error <- build_mess...
#>     h(simpleError(msg, call))

The reason this happens is that targets is treating tar_option_get("envir") as a simple environment when it exports data to parallel workers:

targets/R/class_crew.R

Lines 104 to 167 in 8ab4319

	produce_exports = function(envir, path_store, is_globalenv = NULL) {
	map(names(envir), ~force(envir[[.x]])) # try to nix high-mem promises
	common <- list()
	globals <- list()
	# Avoid the global environment in autometed tests.
	# Covered in semi-automated tests.
	# nocov start
	if (is_globalenv %|||% identical(envir, globalenv())) {
	globals <- as.list(envir, all.names = TRUE)
	which_globals <- fltr(names(globals), ~!is_internal_name(.x, envir))
	globals <- globals[which_globals]
	common$envir <- "globalenv"
	} else {
	# nocov end
	discard <- fltr(names(envir), ~is_internal_name(.x, envir))
	remove(list = discard, envir = envir)
	common$envir <- envir
	}
	common$path_store <- path_store
	common$fun <- tar_runtime$fun
	common$options <- tar_options$export()
	common$envvars <- tar_envvars()
	list(common = common, globals = globals)
	},
	run_worker = function(target) {
	if (self$garbage_collection) {
	gc()
	}
	self$ensure_exports()
	command <- quote(
	targets::target_run_worker(
	target = target,
	envir = envir,
	path_store = path_store,
	fun = fun,
	options = options,
	envvars = envvars
	)
	)
	data <- self$exports$common
	data$target <- target
	globals <- self$exports$globals
	resources <- target$settings$resources$crew
	name <- target_get_name(target)
	target_prepare(
	target = target,
	pipeline = self$pipeline,
	scheduler = self$scheduler,
	meta = self$meta,
	pending = self$controller$saturated(controller = resources$controller)
	)
	self$sync_meta_time()
	self$controller$push(
	command = command,
	data = data,
	globals = globals,
	substitute = FALSE,
	name = name,
	controller = resources$controller,
	scale = TRUE,
	throttle = TRUE,
	seconds_timeout = resources$seconds_timeout
	)
	},

In other words, the envir argument passed to targets::target_run_worker() is pretty much just pkgload::pkg_ns(), and somehow it becomes empty when sent to a worker.

I can isolate this behavior in a much simpler example using callr from the root of the package file system:

pkgload::load_all()
callr::r(
  function(envir) {
    eval(f(), envir = envir)
  },
  args = list(envir = pkgload::pkg_ns())
)
#> Error: 
#> ! in callr subprocess.
#> Caused by error in `f()`:
#> ! could not find function "f"
#> Type .Last.error to see the more details.

If you know of a better way of marshaling package namespaces in a way that works with pkgload::load_all() and does not conflict with non-package environments, then maybe tar_load_all() could become part of targets for people who want pkgload::load_all()-like behavior. Otherwise, I am not sure how to make this work for a package that is not actually installed.

[krlmlr]

Nice summary, thanks! IIUC, remote workers never access the source code but need everything gift-wrapped so that they can just run it? That was new to me.

Let's find a way to make it work, then! tar_load_all() looks like a good compromise to me, but perhaps it's possible to just make load_all() work too, e.g., by searching for .__DEVTOOLS__ in the package namespaces?

[wlandau]

IIUC, remote workers never access the source code but need everything gift-wrapped so that they can just run it?

Yes, exactly. For example, in the cloud pipelines I am running with targets and crew.aws.batch, the Docker containers of the workers do not have access to local files can cannot load_all() themselves.

Let's find a way to make it work, then! tar_load_all() looks like a good compromise to me, but perhaps it's possible to just make load_all() work too, e.g., by searching for .DEVTOOLS in the package namespaces?

._DEVTOOLS__ could be useful for detecting whether tar_option_set("envir") was created by pkgload::load_all(). (Or maybe .__NAMESPACE__ more generally if we want something that will work with both installed and uninstalled packages.) But as far as making pkgload::load_all() "just work"... even if it is possible, it would require runtime conditional logic to choose the defaults of many tar_option_set() options. In my mind, this is too much technical debt to have at a low level. (Related: unlike drake, static branching is not implemented in the package itself. The implementation is a bunch of wrapper functions in tarchetypes. I prefer to add these kinds of features using extra layers rather than deep infrastructure when possible.)

[asadow]

This is a very exciting idea. I frequently bring up a targets project for data exploration and wish I could load_all() instead of writing many library calls.

tar_load_all() however is a bit confusing with tar_load() already for loading targets.

[cpauvert]

Hi,
thanks @wlandau for developing and continuously maintaining {targets}!

I'm still learning, therefore I cannot offer any solutions regarding the topic at hand. However, it made me think of something. During package development, pkgload::load_all() is used regularly to test and evaluate the WIP functions, and when some of these functions are targets factories, examples cannot be ran as targets crash with:

# edited for clarity
! Error running targets::tar_make()
Last error message:
    there is no package called ‘wip_pkg_loaded_locally_but_not_installed’

I looked up the examples of Targetopia packages as well as dicussions, without a clue. I have a feeling that a wrapper like tar_load_all would enable this.

Let me know if this is off-topic and would require its own thread (with more details, reprex etc), I do not want to pollute this discussion but merely showing interest and bringing another angle.

BEst,