"<anonymous>: ... may be used in an incorrect context" with plan(lazy)

[HenrikBengtsson]

Lazy futures may generate annoying warnings on ": ... may be used in an incorrect context", but they seem to be harmless.

Example

Assume:

library(future)

sum_F <- function(x, ...) {
  y %<=% sum(x, ...)
  y
}

x <- c(1:3, NA)

Then, if we use eager or multicore processing:

> sum(x)
[1] NA
> sum(x, na.rm=TRUE)
[1] 6

> plan(eager)
> sum_F(x)
[1] NA
> sum_F(x, na.rm=TRUE)
[1] 6

> plan(multicore)
> sum_F(x)
[1] NA
> sum_F(x, na.rm=TRUE)
[1] 6

everything is as expected. However, if we use lazy evaluation we get a warning:

> plan(lazy)
> sum_F(x)
[1] NA
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'

> sum_F(x, na.rm=TRUE)
[1] 6
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'

On the upside, the result is correct, which indicates that arguments passed via ... are indeed properly passed down.

How can we get rid of these warnings?

[HenrikBengtsson]

Similar example that narrows down the problem:

> library(future)
> lazy_sum  <- function(x, ...) { lazy(sum(x, ...)) }
> x <- c(1:3, NA)

> f <- lazy_sum(x)
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'
> value(f)
[1] NA

> f <- lazy_sum(x, na.rm=TRUE)
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'
> value(f)
[1] 6

And if we do a traceback:

> options(warn=2)
> f <- lazy_sum(x)
Error: (converted from warning) <anonymous>: ... may be used in an incorrect con
text: 'sum(x, ...)'
> traceback()
18: doWithOneRestart(return(expr), restart)
17: withOneRestart(expr, restarts[[1L]])
16: withRestarts({
        .Internal(.signalCondition(simpleWarning(msg, call), msg,
            call))
        .Internal(.dfltWarn(msg, call))
    }, muffleWarning = function() NULL)
15: .signalSimpleWarning("<anonymous>: ... may be used in an incorrect context:
'sum(x, ...)'\n",
        quote(NULL))
14: warning(msg, call. = FALSE)
13: w$warn(paste(paste(w$name, collapse = " : "), ": ", m, loc, "\n",
        sep = ""))
12: w$signal(paste(a, "may be used in an incorrect context:", pasteExpr(e)),
        w)
11: collectUsageArgs(e, w)
10: w$call(e, w)
9: walkCode(body, w)
8: collectUsageFun(name, formals(fun), body(fun), w)
7: collectUsage(fun, enterGlobal = enter)
6: codetools::findGlobals(fun, merge = TRUE)
5: findGlobals_conservative(expr, envir = envir)
4: findGlobals(expr, envir = envir, ..., method = method, tweak = tweak,
       substitute = FALSE, unlist = unlist)
3: globalsOf(expr, envir = envir, tweak = tweakExpression, primitive = FALSE,
       base = FALSE, unlist = TRUE)
2: lazy(sum(x, ...)) at #2
1: lazy_sum(x)

Thus, the problem has to do with the globals package, and not the future package per se.

[HenrikBengtsson]

So,

> expr <- substitute(sum(x, ...))
> globals::findGlobals(expr)
[1] "sum" "x"
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'

and

> expr <- substitute(sum(x, ...))

> globals:::findGlobals_conservative(expr, envir=parent.frame())
[1] "sum" "x"
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'

> globals:::findGlobals_liberal(expr, envir=parent.frame())
[1] "sum" "x"
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'

Continuing,

> expr <- substitute(sum(x, ...))
> fun <- globals:::asFunction(expr)
> fun
function ()
sum(x, ...)
> codetools::findGlobals(fun, merge=TRUE)
[1] "sum" "x"
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'

[HenrikBengtsson]

When using futures, this comes down to whether ... passed is done so within a function that uses ... or not.

For instance, the following is a false-positive warning:

> library(future)
> lazy_sum  <- function(x, ...) { lazy(sum(x, ...)) }
> f <- lazy_sum(1:3)
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)
> value(f)
[1] 6

whereas the follow is a true-positive warning:

> f <- lazy(sum(1:3, ...))
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)
> value(f)
Error in eval(expr, envir, enclos) : '...' used in an incorrect context

[HenrikBengtsson]

Moreover/importantly, arguments passed via ... are not properly handled, which (is most likely) not an issue with eager, lazy, and multicore strategies part of the future package, but are with for instance async's BatchJobs processing strategies, e.g.

> library(async)
> batchjobs_sum  <- function(x, ...) { batchjobs(sum(x, ...)) }

> batchjobs_sum(1:3)
BatchJobsAsyncTask:
Expression:
  sum(x, ...)
Status: 'error', 'started', 'submitted'
Error: 'Error in eval(expr, envir, enclos) : "..." used in an incorrect context
'
Backend:
Job registry:  async979533714
  Number of jobs:  1
  Files dir: C:/Users/hb/braju.com.R/future/.async/async979533714-files
  Work dir: C:/Users/hb/braju.com.R/future
  Multiple result files: FALSE
  Seed: 1017850827
  Required packages: BatchJobs
Cluster functions: 'Local'
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)'

Same for batchjobs_sum(1:3, na.rm=TRUE).

Workaround

Don't pass ... to future evaluators, but instead collect the corresponding arguments before setting up the future. This can be done as for instance,

> batchjobs_sum  <- function(x, ...) { 
  args <- list(x, ...)
  batchjobs(do.call(sum, args)) 
}
> f <- batchjobs_sum(1:3)
> value(f)
[1] 6

and for the original example as:

> library(future)

> sum_F <- function(x, ...) {
  args <- list(x, ...)
  y %<=% do.call(sum, args)
  y
}

> plan(lazy)
> sum_F(1:3)
[1] 6

# ... which also works with async's BatchJobs futures:
> plan(batchjobs)
> sum_F(1:3)
[1] 6
> plan(batchjobs, backend="local")
> sum_F(1:3)
[1] 6
> plan(batchjobs, backend="multicore")
> sum_F(1:3)
[1] 6
> plan(batchjobs, backend=".BatchJobs.R")
> sum_F(1:3)
[1] 6

[HenrikBengtsson]

Quick note: It is possible to check whether ... exist or not by using exists("...", inherits=TRUE). For example:

sum_1 <- function(x, ...) {
  message("Arguments '...' exists: ", exists("...", inherits=TRUE))
  y %<=% { sum(x, ...) }
  y
}


sum_2 <- function(x) {
  message("Arguments '...' exists: ", exists("...", inherits=TRUE))
  y %<=% { sum(x) }
  y
}

sum_3 <- function(x, ...) {
  sum_2 <- function(x) {
    message("Arguments '...' exists: ", exists("...", inherits=TRUE))
    y %<=% { sum(x, ...) }
    y
  }
  sum_2(x)
}

and

plan(lazy)

> sum_1(1:10)
Arguments '...' exists: TRUE
[1] 55
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)' (:3)

> sum_2(1:10)
Arguments '...' exists: FALSE
[1] 55

> sum_3(1:10)
Arguments '...' exists: TRUE
[1] 55
Warning message:
<anonymous>: ... may be used in an incorrect context: 'sum(x, ...)' (:4)

This kind of test should be done by the globals package.

[HenrikBengtsson]

Fixed; warnings on ... may be used in an incorrect context are no longer produced.

library(future)

sum_F <- function(x, ...) {
  y %<=% sum(x, ...)
  y
}

x <- c(1:3, NA)

> plan(lazy)
> sum_F(x)
[1] NA
> sum_F(x, na.rm=TRUE)
[1] 6

and if there's an error in the future expression, then an error thrown, e.g.

library(future)

sum_E <- function(x) {
  y %<=% sum(x, ...)
  y
}

x <- c(1:3, NA)

> plan(lazy)
> sum_E(x)
Error in evaluator(expr, envir = envir, substitute = FALSE, ...) :
  Did you mean to create the future within a function?  Invalid future expression 
tries to use global '...' variables that do not exist: sum(x, ...)
> sum_E(x, na.rm=TRUE)
Error in sum_E(x, na.rm = TRUE) : unused argument (na.rm = TRUE)