NEWS

Package: future.BatchJobs
=========================

Version: 0.17.0-9000 [2021-01-08]

SIGNIFICANT CHANGES:

 * The 'future.BatchJobs' package has been archived on CRAN as of 2021-01-08.
 

Version: 0.17.0 [2021-01-02]

SIGNIFICANT CHANGES:

 * The 'future.BatchJobs' package is deprecated and will soon be archived on
   CRAN in favor of the 'future.batchtools' package.

 * Lazy batchjobs futures only creates the internal BatchJobs registry when
   the future is launched.
 

Version: 0.16.2 [2019-09-27]

NEW FEATURES:

 * Debug messages are now prepended with a timestamp.
 
BUG FIXES:

 * Argument 'workers' could not be a function.

 * Argument 'workers' of type character was silently accepted and effectively
   interpreted as workers = length(workers).


Version: 0.16.1 [2019-01-03]

FIXES:

 * Made internal code agile to upcoming changes in the future package on
   how a captured error is represented.

BUG FIXES:

 * resolve() on a lazy batchtools future would stall and never return.


Version: 0.16.0 [2018-07-09]

NEW FEATURES:

 * The batchjobs_* backends support the handling of the standard output
   as implemented in future (>= 1.9.0).

FIXES:

 * Since the BatchJobs package is no longer maintained it is has not been
   updated according to the new API of the DBI package. This caused several
   warnings to be produced when using BatchJobs futures.  These warnings
   are now suppressed (to avoid all the noise from these warnings).

SOFTWARE QUALITY:

 * Removed internal workaround for BatchJobs (< 1.7).

CODE REFACTORING:

 * Preparing for futures to gather a richer set of results from BatchJobs
   backends.

DEPRECATED AND DEFUNCT:

 * Removed defunct and hidden argument 'args' to BatchJobsFuture.


Version: 0.15.0 [2017-09-10]

NEW FEATURES:

 * The error message for expired BatchJobs futures now include the last few
   lines of the logged output, which sometimes includes clues on why the future
   expired.  For instance, if a TORQUE / PBS job use more than the allocated
   amount of memory it might be terminated by the scheduler leaving the message
   "PBS: job killed: vmem 1234000 exceeded limit 1048576" in the output.

 * print() for BatchtoolsFuture returns the object invisibly.

BUG FIXES:

 * Calling future_lapply() with functions containing globals part of non-default
   packages would when using batchtools futures give an error complaining that
   the global is missing. This was due to updates in future (>= 1.4.0) that
   broke this package.

 * loggedOutput() for BatchtoolsFuture would always return NULL unless an error
   had occurred.

DEPRECATED AND DEFUNCT:

 * Removed defunct batchjobs() and backend() functions.

 * Passing hidden argument 'args' to BatchJobsFuture is now defunct.
 

Version: 0.14.1 [2017-05-30]

DOCUMENTATION:

 * Removing remaining references to 'eager' (now using 'sequential').

SOFTWARE QUALITY:

 * Testing future_lapply() for batchjobs backends.

 * TESTS: No longer testing with (deprecated) 'lazy' or 'eager' backends.
 

Version: 0.14.0 [2017-03-18]

NEW FEATURES:

 * The number of jobs one can add to the queues of HPC schedulers is in
   principle unlimited, which is why the number of available workers for
   such batchjobs_* backends is reported as +Inf.  However, as the number
   of workers is used by future_lapply() to decide how many futures should
   be used to best partition the elements, this means that future_lapply()
   will always use one future per element.  Because of this, it is now
   possible to specify plan(batchjobs_*, workers = n) where 'n' is the
   target number of workers.
 
 * Option 'future.wait.timeout' (replaces 'future.wait.times') specifies
   the maximum waiting time for BatchJobs futures to finish before
   generating a timeout error.

BUG FIX:

 * Futures with globals would give an error when using the developers
   version of BatchJobs (> 1.6) with globals (<= 0.7.1).  Package now
   requires future (>= 1.4.0) which in turn requires globals (>= 0.8.0).

DEPRECATED AND DEFUNCT:

 * Previously deprecated batchjobs() and backend() functions are defunct.
   Instead, use one of the corresponding batchjobs_*() functions.
 

Version: 0.13.1 [2016-10-20]

NEW FEATURES:

 * Added argument 'job.delay' to batchjobs_*() futures for passing
   it as is to BatchJobs::submitJobs() used when launching futures.
   
 * Added argument 'label' to batchjobs_*() futures which are reflected
   in the job name listed by schedulers.  Because of limitation is
   BatchJobs, not all characters in the labels are supported and are
   therefore dropped in the job names.

GLOBAL VARIABLES:

 * GLOBALS: Now globals can be specified explicitly.


Version: 0.13.0 [2016-08-02]

NEW FEATURES:

 * Added argument 'resources' to batchjobs_*() functions for passing
   it to the BatchJobs template (as variable 'resources').
   
 * ROBUSTNESS: value() now launches the future iff not already done.
   Added protection from launching a future more than once.


Version: 0.12.1 [2016-06-26]

DOCUMENTATION:

 * Advising against _multicore BatchJobs_ futures, because there
   is a risk for long waiting times due to starvation.
   This is a limitation of the BatchJobs package.

BUG FIX:

 * Multicore BatchJobs futures are not supported on Solaris Unix and
   now falls back to local BatchJobs futures (as on Windows).
   This is a limitation of the BatchJobs package.


Version: 0.12.0 [2016-06-25]

NEW FEATURES:

 * Added predefined batchjobs_local(), batchjobs_interactive(),
   batchjobs_multicore(), batchjobs_lsf(), batchjobs_openlava(),
   batchjobs_sge(), batchjobs_slurm(), batchjobs_torque() and
   batchjobs_custom() futures.
   
 * Added nbrOfWorkers() for BatchJobs futures.

 * CLEANUP: Now "Loading required package: BatchJobs [...]", which is
   outputted when the first BatchJobs future is created, is suppressed.

DEPRECATED AND DEFUNCT:

 * Removed non-used completed(), failed() and expired() for BatchJobs objects.

 * Deprecated plan(batchjobs, backend=...).

 * Deprecated backend().


Version: 0.11.0 [2016-05-16]

NEW FEATURES:

 * backend("multicore=1") or other multicore specifications that
   result in single-core processing will use backend("local") instead.

 * WORKAROUND: The BatchJobs multicore cluster functions are designed
   to give some leeway for other processes on the local machine.
   Unfortunately, this may result in endless or extremely long waiting
   for free resources before BatchJobs multicore jobs can be submitted.
   One reason is that BatchJobs tries to keep the average CPU load
   below a threshold that is calculated based on the number of cores.
   Unfortunately, this can result in starvation due to other processes,
   especially if the number of cores on the machine is small and/or if
   mc.cores is set to a small number.  Because of this, we disable this
   mechanism (by using BatchJobs parameter max.load=+Inf).

 * Now BatchJobsFutureError extends FutureError.
   
DOCUMENTATION:

 * Add package vignette.

BUG FIX:

 * BUG FIX: backend("multicore-3") was interpreted as backend("multicore").


Version: 0.10.0 [2016-05-03]

NEW FEATURES:

 * Now the BatchJobsFutureError records the captured BatchJobs
   output to further simplify post mortem troubleshooting.
   
 * delete() for BatchJobsFuture will no longer remove the BatchJobs
   registry files if the BatchJobs has status 'error' or 'expired'
   and (new) option 'future.delete' is not set to FALSE (which it
   is if running in interactive mode).  The new setup is useful
   for troubleshooting failed BatchJobs futures in non-interactive
   R sessions, which otherwise would be cleaned out when the R
   session terminates (due to garbage collection calling delete()).

BUG FIX:

 * resolved() on a BatchJobs future could return FALSE even after
   value() was called.  Added package test.


Version: 0.9.0 [2016-04-15]

NEW FEATURES:

 * Package renamed to future.BatchJobs (was async).

 * Package requires R (>= 3.2.0) just so Mandelbrot demo works.

 * STANDARIZATION Now using option and environment names already defined
   by the future package, i.e. future.maxTries, future.interval, and
   R_FUTURE_MAXTRIES (used to be named async::* and R_ASYNC_*).
   
 * STANDARIZATION: Directories for BatchJobs are now created under
   .future/<session>/ of the current directory (was .async/<session>/).
   Also, those subdirectories now use prefix 'BatchJobs_' (was 'async').
   This was done to have a common directory structure also for other
   future backends that needs to keep files on the file system.

DEPRECATED AND DEFUNCT:

 * CLEANUP: Renamed AsyncTaskError to BatchJobsFutureError.

 * CLEANUP: Dropping AsyncListEnv.


Version: 0.8.0 [2016-04-14]

NEW FEATURES:

 * batchjobs() function gained class attribute.
 
 * Renamed BatchJobsAsyncTask to BatchJobsFuture.
 
 * CLEANUP: Removed no-longer needed asyncBatchEvalQ() because
   BatchJobsFuture is now self sufficient.

BUG FIX:

 * Global variables with the same name as objects in the base or the
   BatchJobs package would be overridden by the latter, e.g. a global
   variable 'col' would be masked by 'base::col'. (Issue #55)


Version: 0.7.1 [2016-01-04]

BUG FIX:

 * New BatchJobs work directories would encode 08:03 as ' 803' instead
   of '0803' resulting in a BatchJobs assertion error on invalid pathnames.


Version: 0.7.0 [2016-01-02]

NEW FEATURES:

 * Now value() for BatchJobsAsyncTask removes associated BatchJobs
   subdirectories upon success.  Previously, such cleanup was only
   happening when the object was garbage collected.
   
 * Each R session that load the async package now uses a unique
   subdirectory under .async/, e.g. .async/20160102_154202-IVBRy1/.
   It is in turn under that session-specific subdirectory that the
   individual BatchJobs subdirectories corresponding to a specific
   future lives.  Note that, although, the each of latter is removed
   when calling value() for its future, the session-specific async
   directory is not removed.  In order to remove the latter, make
   sure to resolve all futures.  Then call unloadNamespace("async"),
   which will try to remove the directory.


Version: 0.6.2 [2015-11-21]

BUG FIX:

 * asyncBatchEvalQ() would not export globals that belongs to a package
   but are not exported.


Version: 0.6.1 [2015-10-20]

NEW FEATURES:

 * CLEANUP: Package no longer attaches listenv.

BUG FIX:

 * Globals that were copies of package objects were not exported to the
   future environments.


Version: 0.6.0 [2015-10-05]

GLOBAL VARIABLES:

 * batchjobs(sum(x, ...), globals=TRUE) now handles `...` properly.

 * ROBUSTNESS: asyncBatchEvalQ() gives an informative error when
   a global variables starting with a period needs to be exported;
   these are currently not supported due to limitations in the
   BatchJobs package.

BUG FIX:

 * resolved() for AsyncFuture:s would always give FALSE unless value()
   of the future has been called first.

 * WORKAROUND: Global variables with names starting with a period
   or that does not match pattern '[a-zA-Z0-9._-]+' could not be
   exported due to BatchJobs limitation.  Until resolved by BatchJobs,
   this package encode and decode such variable names automatically.


Version: 0.5.2 [2015-07-30]

DEPRECATED AND DEFUNCT:

 * CLEANUP: Dropped %backend% - use %plan% backend(...) instead.

BUG FIX:

 * batchjobs(..., backend="interactive") changed also the default backend.


Version: 0.5.1 [2015-07-29]

 * Adjusted to future (>= 0.7.0).

 * CLEANUP: Dropped functions and tests that are now in the
   future package.


Version: 0.5.0 [2015-06-19]

 * Adjusted to future (>= 0.5.1).


Version: 0.4.2 [2015-06-14]

NEW FEATURES:

 * Added run() for BatchJobsAsyncTask.

DOCUMENTATION:

 * Added demo("mandelbrot", package="async").

DEPRECATED AND DEFUNCT:

 * CLEANUP: BatchJobsAsyncTask no longer registers/submits jobs.

 * CLEANUP: Dropped asyncEvalQ().

 * CLEANUP: Dropped async() - now batchjobs().
 
 * CLEANUP: Dropped makeClusterFunctionsRscript().
 
 * CLEANUP: Dropped delayed assignment %<-% infix operator.


Version: 0.4.1 [2015-06-14]

NEW FEATURES:

 * Add batchjobs() allowing for plan(batchjobs, backend="multicore").

 * BatchJobsAsyncTask() and internal tempRegistry() gained
   argument 'backend'.


Version: 0.4.0 [2015-06-08]

 * CLEANUP: Extract Future API and moved to new package 'future'.

 * Now delayedAsyncAssign() returns a Future.

BUG FIX:

 * The existence of .BatchJobs.R would override whatever backend was
   already set by backend().
   
 * Asynchronous evaluation of { a <<- 1 } no longer identifies 'a' as
   a global variable that needs to be exported.


Version: 0.3.1 [2015-05-23]

 * CLEANUP: Moved more internal code to the 'listenv' package.


Version: 0.3.0 [2015-05-21]

NEW FEATURES:

 * Now inspect(envir=x) returns all tasks if only the environment
   is specified, e.g. inspect(envir=x) vs inspect(x$a).

 * Added completed() and failed(), expired().

 * Any flavor of backend("multicore") is based on availableCores().

 * Added availableCores() for identifying the number of available cores.
   The default is to acknowledged the assigned number of cores by
   queueing systems such as Torque/PBS, before using detectCores() of
   the 'parallel' package.

 * ROBUSTNESS: Asynchronous tasks that still run when R exists
   will not be stopped and not deleted.  This will allow the tasks
   running on job clusters to complete.

 * CLEANUP: Moved list environments to new 'listenv' package.

 * CLEANUP: Moved identification of globals to new 'globals' package.

BUG FIX:

 * AsyncTask objects were not assigned to the listenv.


Version: 0.2.0 [2015-05-11]

NEW FEATURES:

 * Functions AsyncTask() and delayedAsyncAssign() gained argument
   'substitute' for controlling whether the expression/value
   should  be substitute():d or not.

 * ROBUSTNESS: Added package tests for delayedAsyncAssign().

 * CLEANUP: Internal restructuring with more informative classes.


Version: 0.1.4 [2015-05-02]

NEW FEATURES:

 * Added print() for listenv:s.

 * CLEANUP: Using tempvar() of R.utils.


Version: 0.1.3 [2015-04-26]

NEW FEATURES:

 * Added AsyncListEnv.

GLOBAL VARIABLES:

 * ROBUSTNESS: Add protection for trying to evaluating asynchronous
   expressions with global objects that are "too large" and therefore
   introduces lots of overhead in exporting to, and importing from
   workers.  The size limit of the maximum allowed total export size
   is controlled by option 'async::maxSizeOfGlobals'.


Version: 0.1.2 [2015-04-21]

NEW FEATURES:

 * Now status(), finished() etc. for AsyncTask returns NA in case
   task backend registry is deleted.  print() does a better job
   too in this case.

 * Now inspect() also accepts complex input such as inspect(a$x),
   inspect(a[["x"]]) and inspect(a[[1]]).  It also accepts a
   character name such as inspect("x", envir=a).

 * Now await() for AsyncTask gives an more informative error message
   in case the backend registry was preemptively deleted.

 * Added error classes AsyncError and AsyncTaskError with more
   informative error messages simplifying troubleshooting.

 * CLEANUP: Now async BatchJobs registries are created in ./.async/

BUG FIX:

 * Delayed (synchronous and asynchronous) assignments to listenv:s did
   not update the internal name-to-variable map, which effectively made
   such listenv:s object empty (although the assign value was stored
   internally).


Version: 0.1.1 [2015-04-07]

BUG FIX:

 * asyncBatchEvalQ() would given "Error in packageVersion(pkg) : package
   'R_GlobalEnv'" if the expression had a global function defined in the
   global environment.  Now asyncBatchEvalQ() does a better jobs in
   identifying package names.  Added package tests for this case.


Version: 0.1.0 [2015-02-07]

 * First prototype of an old idea of asynchronous evaluations with
   delayed assignments.

 * Created.