/
NEWS
1561 lines (1040 loc) · 58.7 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Package: future
===============
Version: 1.16.0-9000 [2020-03-31]
SIGNIFICANT CHANGES:
* Renamed values() to value() to cleanup and simplify the API.
NEW FEATURES:
* makeClusterPSOCK() gained argument 'rscript_envs' for setting environment
variables in workers on startup, e.g. rscript_envs = c(FOO = "3.14", "BAR").
* Now the result of a future holds session details in case an error occurred
while evaluating the future.
MISCELLANEOUS:
* Not all CRAN servers have _R_CHECK_LIMIT_CORES_ set. To better emulate
CRAN submission checks, the future package will, when loaded, set this
environment variable to 'TRUE' if unset and if 'R CMD check' is running.
Note that future::availableCores() respects _R_CHECK_LIMIT_CORES_ and
returns at most 2L (two cores) if detected.
BUG FIXES:
* Disabling of multi-threading when using 'multicore' futures did not work
on all platforms.
DEPRECATED AND DEFUNCT:
* All values() S3 methods have been renamed to value() since they are closely
related to the original purpose value(). The values() methods will continue
to work but will soon be formally deprecated and later be made defunct and
finally be removed. Please replace all values() with value() calls.
Version: 1.16.0 [2020-01-15]
SIGNIFICANT CHANGES:
* Now oplan <- plan(new_strategy) returns the list of all nested strategies
previously set, instead of just the strategy on top of this stack. This
makes it easier to temporarily use another plan. For the old behavior,
use 'oplan <- plan(new_strategy)[[1]]'.
NEW FEATURES:
* Now value() detects if a future(..., seed = FALSE) call generated random
numbers, which then might give unreliable results because non-parallel
safe, non-statistically sound random number generation (RNG) was used.
If option 'future.rng.onMisuse' is "warning", a warning is produced. If
"error", an error is produced. If "ignore" (default), the mistake is
silently ignored. Using seed = NULL is like seed = FALSE but without
performing the RNG validation.
* For convenience, argument 'seed' of future() may now also be an ordinary
single integer random seed. If so, a L'Ecuyer-CMRG RNG seed is created
from this seed. If seed = TRUE, then a L'Ecuyer-CMRG RNG seed based on the
current RNG state is used. Use seed = FALSE when it is known that the
future does not use RNG.
* ClusterFuture:s now relay immediateCondition:s back to the main process
momentarily after they are signaled and before the future is resolved.
BETA FEATURES:
* Add support for automatically disable multi-threading when using 'multicore'
futures. For now, the default is to allow multi-threaded processing but
this might change in the future. To disable multi-threaded, set option
'future.fork.multithreading.enable' or environment variable
'R_FUTURE_FORK_MULTITHREADING_ENABLE' to 'FALSE'. This requires that
'RhpcBLASctl' package is installed. Parallelization via multi-threaded
processing (done in native code by some packages and externaly library)
while at the same time using forked (aka "multicore") parallel processing
is unstable in some cases. Note that this is not only true when using
plan(multicore) but also when using, for instance, parallel::mclapply().
This is in beta so the above names and options might change later.
BUG FIXES:
* Evaluation of futures could fail if the global environment contained
*functions* with the same names as a small set of base R functions, e.g.
'raw()', 'list()', and 'options()'.
* future(alist(a=)) would produce "Error in objectSize_list(x, depth = depth
- 1L) : argument "x_kk" is missing, with no default"
DEPRECATED AND DEFUNCT:
* Future and FutureResult objects with an internal version 1.7 or older have
been deprecated since 1.14.0 (July 2019) and are now defunct.
* Defunct hidden argument 'progress' of resolve(), and hidden arguments/fields
'condition' and 'calls' of FutureResult are now gone.
Version: 1.15.1 [2019-11-23]
NEW FEATURES:
* The default range of ports that makeClusterPSOCK() draws a random port from
(when argument 'port' is not specified) can now be controlled by environment
variable `R_FUTURE_RANDOM_PORTS`. The default range is still 11000:11999
as with the 'parallel' package.
BUG FIXES:
* The change introduced to resolved() in future 1.15.0 would cause lazy
futures to block if all workers were occupied.
Version: 1.15.0 [2019-11-07]
SIGNIFICANT CHANGES:
* resolved() will now launch lazy futures.
NEW FEATURES:
* Now the "visibility" of future values is recorded and reflected by value().
* Now option 'future.globals.onReference' defaults to environment variable
'R_FUTURE_GLOBALS_ONREFERENCE'.
DOCUMENTATION:
* Added 'Troubleshooting' section to ?makeClusterPSOCK with instructions on
how to troubleshoot when the setup of local and remote clusters fail.
BUG FIXES:
* values() would resignal immediateCondition:s despite those should only be
signalled at most once per future.
* makeClusterPSOCK() could produce warnings like "cannot open file
'/tmp/alice/Rtmpi69yYF/future.parent=2622.a3e32bc6af7.pid': No such file",
e.g. when launching R workers running in Docker containers.
* Package would set or update the RNG state of R (.Random.seed) when loaded,
which could affect RNG reproducibility.
* Package could set '.Random.seed' to NULL, instead of removing it, which in
turn would produce a warning on "'.Random.seed' is not an integer vector but
of type 'NULL', so ignored" when the next random number generated.
* Now a future assignment to list environments produce more informative error
messages if attempting to assign to more than one element.
* makeClusterMPI() did not work for MPI clusters with 'comm' other than '1'.
DEPRECATED AND DEFUNCT:
* Argument 'value' of resolve() is deprecated. Use 'result' instead.
* Usage of internal argument 'evaluator' is now defunct.
Version: 1.14.0 [2019-07-01]
SIGNIFICANT CHANGES:
* All types of conditions are now captured and relayed. Previously, only
conditions of class 'message' and 'warning' were relayed.
* If one of the futures in a collection produces an error, then values()
will signal that error as soon as it is detected. This means that while
calling values() guarantees to resolve all futures, it does not guarantee
that the result from all futures are gathered back to the master R session
before the error is relayed.
NEW FEATURES:
* values() now relays 'stdout' and 'signal' as soon as possible as long as
the standard output and the conditions are relayed in their original order.
* If a captured condition can be "muffled", then it will be muffled. This
helps to prevent conditions from being handled twice by condition handlers
when futures are evaluated in the main R session, e.g. plan(sequential).
Messages and warnings were already muffled in the past.
* Forked processing is considered unstable when running R from certain
environments, such as the RStudio environment. Because of this, 'multicore'
futures have been disabled in those cases since future 1.13.0. This change
caught several RStudio users by surprise. Starting with future 1.14.0, an
informative one-time-per-session warning will be produced when attempts to
use 'multicore' is made in non-supported environments such as RStudio.
This warning will also be produced when using 'multiprocess', which will
fall back to using 'multisession' futures. The warning can be disabled by
setting R option 'future.supportsMulticore.unstable', or environment
variable 'FUTURE_SUPPORTSMULTICORE_UNSTABLE' to "quiet".
* Now option 'future.startup.script' falls back to environment variable
'R_FUTURE_STARTUP_SCRIPT'.
* Conditions inheriting 'immediateCondition' are signaled as soon as possible.
Contrary to other types of conditions, these will be signaled only once per
future, despite being collected.
BUG FIXES:
* Early signaling did not take place for resolved() for ClusterFuture and
MulticoreFuture.
* When early signaling was enabled, functions such as resolved() and
resolve() would relay captured conditions multiple times. This would,
for instance, result in the same messages and warnings being outputted
more than once. Now it is only value() that will resignal conditions.
* The validation of connections failed to detect when the connection had
been serialized (= a NIL external pointer) on some macOS systems.
DEPRECATED AND DEFUNCT:
* Argument 'progress' of resolve() is now defunct (was deprecated since
future 1.12.0). Option 'future.progress' is ignored. This will make
room for other progress-update mechanisms that are in the works.
* Usage of internal argument 'evaluator' is now deprecated.
* Removed defunct argument 'output' from FutureError().
* FutureResult fields/arguments 'condition' and 'calls' are now defunct.
Use 'conditions' instead.
* Future and FutureResult objects with an internal version 1.7 or older are
deprecated and will eventually become defunct. Future backends that
implement their own Future classes should update to implement a result()
method instead of a value() method for their Future classes. All future
backends available on CRAN and Bioconductor have already been updated
accordingly.
Version: 1.13.0 [2019-05-08]
SIGNIFICANT CHANGES:
* Forked processing is now disabled by default when running R via RStudio
When disabled, 'multicore' futures fall back to a 'sequential' futures.
This update follows from an RStudio recommendation against using _forked_
parallel processing from within RStudio because it is likely to break the
RStudio R session. See help("supportsMulticore") for more details, e.g.
how to re-enable process forking. Note that parallelization via
'multisession' is unaffected and will still work as before. Also, when
forked processing is disabled, or otherwise not supported, using
plan("multiprocess") will fall back to using 'multisession' futures.
NEW FEATURES:
* Forked processing can be disabled by setting R option 'future.fork.enable'
to FALSE (or environment variable 'R_FUTURE_FORK_ENABLE=false'). When
disabled, 'multicore' futures fall back to a 'sequential' futures even if
the operating system supports process forking. If set of TRUE, 'multicore'
will not fall back to 'sequential'. If NA, or not set (the default), a
set of best-practices rules will decide whether forking is enabled or not.
See help("supportsMulticore") for more details.
* Now availableCores() also recognizes PBS environment variable 'NCPUS',
because the PBSPro scheduler does not set 'PBS_NUM_PPN'.
* If, option 'future.availableCores.custom' is set to a function, then
availableCores() will call that function and interpret its value as
number of cores. Analogously, option 'future.availableWorkers.custom'
can be used to specify a hostnames of a set of workers that
availableWorkers() sees. These new options provide a mechanism for anyone
to customize availableCores() and availableWorkers() in case they do not
(yet) recognize, say, environment variables that are specific the user's
compute environment or HPC scheduler.
* makeClusterPSOCK() gained support for argument 'rscript_startup' for
evaluating one or more R expressions in the background R worker prior
to the worker event loop launching. This provides a more convenient
approach than having to use, say, 'rscript_args = c("-e", sQuote(code))'.
* makeClusterPSOCK() gained support for argument 'rscript_libs' to control the
R package library search path on the workers. For example, to _prepend_ the
folder '~/R-libs' on the workers, use 'rscript_libs = c("~/R-libs", "*")',
where "*" will be resolved to the current '.libPaths()' on the workers.
* Debug messages are now prepended with a timestamp.
DOCUMENTATION:
* Add vignette on 'Non-Exportable Objects' (extracted from another vignette).
BUG FIXES:
* makeClusterPSOCK() did not shell quote the Rscript executable when running
its pre-tests checking whether localhost Rscript processes can be killed by
their PIDs or not.
DEPRECATED AND DEFUNCT:
* Argument 'value' of resolve() has been renamed to `result` to better reflect
that not only values are collected when this argument is used. Argument
'value' still works for backward compatibility, but will eventually be
formally deprecated and then defunct.
Version: 1.12.0 [2019-03-07]
NEW FEATURES:
* If makeClusterPSOCK() fails to create one of many nodes, then it will
attempt to stop any nodes that were successfully created. This lowers
the risk for leaving R worker processes behind.
* Future results now hold the timestamps when the evaluation of the future
started and finished.
BUG FIXES:
* Functions no longer produce "partial match of 'condition' to 'conditions'"
warnings with options(warnPartialMatchDollar=TRUE).
* When future infix operators (%conditions%, %globals%, %label%, %lazy%,
%packages%, %seed%, and %stdout%) that are intended for future assignments
were used in the wrong context, they would incorrectly be applied to the
next future created. Now they're discarded.
* makeClusterPSOCK() in future (>= 1.11.1) produced warnings when argument
'rscript' had length(rscript) > 1.
* Validation of L'Ecuyer-CMRG RNG seeds failed in recent R devel.
* With options(OutDec = ","), the default value of several argument would
resolve to NA_real_ rather than a numeric value resulting in errors such
as "is.finite(alpha)’ is not TRUE".
DEPRECATED AND DEFUNCT:
* Argument 'progress' of resolve() is now deprecated.
* Argument 'output' of FutureError() is now defunct.
* FutureError no longer inherits simpleError.
Version: 1.11.1.1 [2019-01-25]
BUG FIXES:
* When makeClusterPSOCK() fails to connect to a worker, it produces an error
with detailed information on what could have happend. In rare cases,
another error could be produced when generating the information on what
the workers PID is.
Version: 1.11.1 [2019-01-25]
NEW FEATURES:
* The defaults of several arguments of makeClusterPSOCK() and makeNodePSOCK()
can now be controlled via environment variables in addition to R options
that was supported in the past. An advantage of using environment variables
is that they will be inherited by child processes, also nested ones.
* The printing of future plans is now less verbose when the 'workers' argument
is a complex object such as a PSOCK cluster object. Previously, the output
would include verbose output of attributes etc.
SOFTWARE QUALITY:
* TESTS: When the 'future' package is loaded, it checks whether 'R CMD check'
is running or not. If it is, then a few future-specific environment
variables are adjusted such that the tests play nices with the testing
environment. For instance, it sets the socket connection timeout for
PSOCK cluster workers to 120 seconds (instead of the default 30 days!).
This will lower the risk for more and more zombie worker processes
cluttering up the test machine (e.g. CRAN servers) in case a worker process
is left behind despite the main R processes is terminated. Note that these
adjustments are applied automatically to the checks of any package that
depends on, or imports, the 'future' package.
BUG FIXES:
* Whenever makeClusterPSOCK() would fail to connect to a worker, for instance
due to a port clash, then it would leave the R worker process running - also
after the main R process terminated. When the worker is running on the same
machine, makeClusterPSOCK() will now attempt to kill such stray R processes.
Note that parallel::makePSOCKcluster() still has this problem.
Version: 1.11.0 [2019-01-21]
SIGNIFICANT CHANGES:
* Message and warning conditions are now captured and relayed by default.
NEW FEATURES:
* The future call stack ("traceback") is now recorded when the evaluation of a
future produces an error. Use backtrace() on the future to retrieve it.
* Now futureCall() defaults to args = list() making is easier to call
functions that do not take arguments, e.g. futureCall(function() 42).
* plan() gained argument '.skip = FALSE'. When TRUE, setting the same future
trategy as already set will be skipped, e.g. calling plan(multisession)
consecutively will have the same effect as calling it just once.
* makeClusterPSOCK() produces more informative error messages whenever the
setup of R workers fails. Also, its verbose messages are now prefixed
with "[local output] " to help distinguish the output produced by the
current R session from that produced by background workers.
* It is now possible to specify what type of SSH clients makeClusterPSOCK()
automatically searches for and in what order, e.g.
'rshcmd = c("<rstudio-ssh>", "<putty-plink>")'.
* Now makeClusterPSOCK() preserves the global RNG state (.Random.seed) also
when it draws a random port number.
* makeClusterPSOCK() gained argument 'rshlogfile'.
* Cluster futures provide more informative error messages when the
communication with the worker node is out of sync.
BUG FIXES:
* Argument 'stdout' was forced to TRUE when using single-core multicore or
single-core multisession futures.
* When evaluated in a local environment, futureCall(..., globals = "a") would
set the value of global 'a' to NULL, regardless if it exists or not and
what its true value is.
* makeClusterPSOCK(..., rscript = "my_r") would in some cases fail to find
the intended 'my_r' executable.
* ROBUSTNESS: A cluster future, including a multisession one, could retrieve
results from the wrong workers if a new set of cluster workers had been
set up after the future was created/launched but before the results were
retrieved. This could happen because connections in R are indexed solely
by integers which are recycled when old connections are closed and new ones
are created. Now cluster futures assert that the connections to the workers
are valid, and if not, an informative error message is produced.
* Calling result() on a non-resolved UniprocessFuture would signal evaluation
errors.
DEPRECATED AND DEFUNCT:
* Removed defunct future::future_lapply(). Please use the one in the
future.apply package instead.
Version: 1.10.0 [2018-10-16]
NEW FEATURES:
* Add support for manually specifying globals in addition to those that are
automatically identified via argument 'globals' or %globals%. Two examples
are globals = structure(TRUE, add = list(a = 42L, b = 3.14)) and
globals = structure(TRUE, add = c("a", "b")). Analogously, attribute
'ignore' can be used to exclude automatically identified globals.
* The error reported when failing to retrieve the results of a future
evaluated on a localhost cluster/multisession worker or a forked/multicore
worker is now more informative. Specifically, it mentions whether the worker
process is still alive or not.
* Add makeClusterMPI(n) for creating MPI-based clusters of a similar kind as
parallel::makeCluster(n, type = "MPI") but that also attempts to workaound
issues where parallel::stopCluster() causes R to stall.
* makeClusterPSOCK() and makeClusterMPI() gained argument 'autoStop' for
controlling whether the cluster should be automatically stopped when
garbage collected or not.
* BETA: Now resolved() for ClusterFuture is non-blocking also for clusters
of type MPIcluster as created by parallel::makeCluster(..., type = "MPI").
BUG FIXES:
* On Windows, plan(multiprocess) would not initiate the workers. Instead
workers would be set up only when the first future was created.
Version: 1.9.0 [2018-07-22]
SIGNIFICANT CHANGES:
* Standard output is now captured and re-outputted when value() is called.
This new behavior can be controlled by the argument 'stdout' to future()
or by specifying the %stdout% operator if a future assignment is used.
NEW FEATURES:
* R option 'width' is passed down so that standard output is captured
consistently across workers and consistently with the master process.
* Now more 'future.*' options are passed down so that they are also
acknowledged when using nested futures.
DOCUMENTATION:
* Add vignette on 'Outputting Text'.
* CLEANUP: Only the core parts of the API are now listed in the help index.
This was done to clarify the Future API. Help for non-core parts are still
via cross references in the indexed API as well via help().
BUG FIXES:
* When using forced, nested 'multicore' parallel processing, such as,
plan(list(tweak(multicore, workers = 2), tweak(multicore, workers = 2))),
then the child process would attempt to resolve futures owned by the parent
process resulting in an error (on 'bad error message').
* When using plan(multicore), if a forked worker would terminate unexpectedly,
it could corrupt the master R session such that any further attempts of
using forked workers would fail. A forked worker could be terminated this
way if the user pressed Ctrl-C (the worker receives a SIGINT signal).
* makeClusterPSOCK() produced a warning when environment variable
'R_PARALLEL_PORT' was set to 'random' (e.g. as on CRAN).
* Printing a plan() could produce an error when the deparsed call used to set
up the plan() was longer than 60 characters.
DEPRECATED AND DEFUNCT:
* future::future_lapply() is defunct (gives an error if called). Please use
the one in the future.apply package instead.
* Argument 'output' of FutureError() is formally deprecated.
* Removed all FutureEvaluationCondition classes and related methods.
Version: 1.8.1 [2018-05-02]
NEW FEATURES:
* getGlobalsAndPackages() gained argument 'maxSize'.
* makeClusterPSOCK() now produces a more informative warning if environment
variable R_PARALLEL_PORT specifies a non-numeric port.
* Now plan() gives a more informative error message in case it fails, e.g.
when the internal future validation fails and why.
* Added UnexpectedFutureResultError to be used by backends for signalling
in a standard way that an unexpected result was retrieved from a worker.
BUG FIXES:
* When the communication between an asynchronous future and a background R
process failed, further querying of the future state/results could end up
in an infinite waiting loop. Now the failed communication error is
recorded and re-signalled if any further querying attempts.
* Internal, seldom used myExternalIP() failed to recognize IPv4 answers
from some of the lookup servers. This could in turn produce another error.
* In R (>= 3.5.0), multicore futures would produce multiple warnings
originating from querying whether background processes have completed
or not. These warnings are now suppressed.
Version: 1.8.0 [2018-04-08]
SIGNIFICANT CHANGES:
* Errors produces when evaluating futures are now (re-)signaled on the master
R process "as is" with the original content and class attributes.
NEW FEATURES:
* More errors related to orchestration of futures are of class FutureError to
make it easier to distinguish them from future evaluation errors.
* Add support for a richer set of results returned by resolved futures.
Previously only the value of the future expression, which could be a
captured error to be resignaled, was expected. Now a FutureResult object
may be returned instead. Although not supported in this release, this
update opens up for reporting on additional information from the evaluation
of futures, e.g. captured output, timing and memory benchmarks etc. Before
that can take place, existing future backend packages will have to be
updated accordingly.
* backtrace() returns only the last call that produced the error. It is
unfortunately not possible to capture the call stack that led up to the
error when evaluating a future expression.
BUG FIXES:
* value() for MulticoreFuture would not produce an error when a (forked)
background R workers would terminate before the future expression is
resolved. This was a limitation inherited from the parallel package.
Now an informative FutureError message is produced.
* value() for MulticoreFuture would not signal errors unless they inherited
from 'simpleError' - now it's enough for them to inherits from 'error'.
* value() for ClusterFuture no longer produces a FutureEvaluationError, but
FutureError, if the connection to the R worker has changed (which happens
if something as drastic as closeAllConnections() have been called.)
* futureCall(..., globals = FALSE) would produce "Error: second argument must
be a list", because the explicit arguments where not exported. This could
also happen when specifying globals by name or as a named list.
* Nested futures were too conservative in requiring global variables to exist,
even when they were false positives.
DEPRECATED AND DEFUNCT:
* future::future_lapply() is formally deprecated. Please use the one in the
future.apply package instead.
* Recently introduced FutureEvaluationCondition classes are deprecated,
because they no longer serve a purpose since future evaluation conditions
are now signaled as is.
Version: 1.7.0 [2018-02-10]
SIGNIFICANT CHANGES:
* future_lapply() has moved to the future.apply package available on CRAN.
NEW FEATURES:
* Argument 'workers' of future strategies may now also be a function, which
is called without argument when the future strategy is set up and used as
is. For instance, plan(multiprocess, workers = halfCores) where
halfCores <- function() { max(1, round(availableCores() / 2)) } will use
half of the number of available cores. This is useful when using nested
future strategies with remote machines.
* On Windows, makeClusterPSOCK(), and therefore plan(multisession) and
plan(multiprocess), will use the SSH client distributed with RStudio as
a fallback if neither 'ssh' nor 'plink' is available on the system PATH.
* Now plan() makes sure that nbrOfWorkers() will work for the new strategy.
This will help catch mistakes such as plan(cluster, workers = cl) where
'cl' is a basic R list rather than a 'cluster' list early on.
* Added %packages% to explicitly control packages to be attached when a
future is resolved, e.g. y %<-% { YT[2] } %packages% "data.table".
Note, this is only needed in cases where the automatic identification of
global and package dependencies is not sufficient.
* Added condition classes FutureCondition, FutureMessage, FutureWarning, and
FutureError representing conditions that occur while a future is setup,
launched, queried, or retrieved. They do *not* represent conditions that
occur while evaluating the future expression. For those conditions, new
classes FutureEvaluationCondition, FutureEvaulationMessage,
FutureEvaluationWarning, and FutureEvaluationError exists.
DOCUMENTATION:
* Vignette 'Common Issues with Solutions' now documents the case where the
future framework fails to identify a variable as being global because it
is only so conditionally, e.g. 'if (runif(1) < 1/2) x <- 0; y <- 2 * x'.
BETA FEATURES:
* Added mechanism for detecting globals that _may_ not be exportable to an
external R process (a "worker"). Typically, globals that carry connections
and external pointers ("externalptr") can not be exported, but there are
exceptions. By setting options 'future.globals.onReference' to "warning",
a warning is produced informing the user about potential problems. If
"error", an error is produced. Because there might be false positive, the
default is "ignore", which will cause above scans to be skipped. If there
are non-exportable globals and these tests are skipped, a run-time error
may be produced only when the future expression is evaluated.
BUG FIXES:
* The total size of global variables was overestimated, and dramatically so
if defined in the global environment and there were are large objects there
too. This would sometimes result in a false error saying that the total
size is larger than the allowed limit.
* An assignment such as 'x <- x + 1' where the left-hand side (LHS) 'x' is a
global failed to identify 'x' as a global because the right-hand side (RHS)
'x' would override it as a local variable. Updates to the globals package
fixed this problem.
* makeClusterPSOCK(..., renice = 19) would launch each PSOCK worker via
'nice +19' resulting in the error "nice: '+19': No such file or directory".
This bug was inherited from parallel::makePSOCKcluster(). Now using
'nice --adjustment=19' instead.
* Protection against passing future objects to other futures did not work
for future strategy 'multicore'.
DEPRECATED AND DEFUNCT:
* future_lapply() has moved to the new future.apply package available on CRAN.
The future::future_lapply() function will soon be deprecated, then defunct,
and eventually be removed from the future package. Please update your code
to make use of future.apply::future_lapply() instead.
* Dropped defunct 'eager' and 'lazy' futures; use 'sequential' instead.
* Dropped defunct arguments 'cluster' and 'maxCores'; use 'workers' instead.
* In previous version of the future package the FutureError class was used
to represent both orchestration errors (now FutureError) and evaluation
errors (now FutureEvaluationError). Any usage of class FutureError for
the latter type of errors is deprecated and should be updated to
FutureEvaluationError.
Version: 1.6.2 [2017-10-16]
NEW FEATURES:
* Now plan() accepts also strings such as "future::cluster".
* Now backtrace(x[[ER]]) works also for non-environment 'x':s, e.g. lists.
BUG FIXES:
* When measuring the size of globals by scanning their content, for certain
types of classes the inferred lengths of these objects were incorrect
causing internal subset out-of-range issues.
* print() for Future would output one global per line instead of
concatenating the information with commas.
Version: 1.6.1 [2017-09-08]
NEW FEATURES:
* Now exporting getGlobalsAndPackages().
BUG FIXES:
* future_lapply() would give "Error in objectSize.env(x, depth = depth - 1L):
object 'nnn' not found" when for instance 'nnn' is part of an unresolved
expression that is an argument value.
SOFTWARE QUALITY:
* FIX: Some of the package assertion tests made too precise assumptions about
the object sizes, which fails with the introduction of ALTREP in R-devel
which causes the R's SEXP header size to change.
Version: 1.6.0 [2017-08-11]
NEW FEATURES:
* Now tweak(), and hence plan(), generates a more informative error message if
a non-future function is specified by mistake, e.g. calling plan(cluster)
with the 'survival' package attached after 'future' is equivalent to calling
plan(survival::cluster) when plan(future::cluster) was intended.
BUG FIXES:
* nbrOfWorkers() gave an error with plan(remote). Fixed by making the 'remote'
future inherit 'cluster' (as it should).
SOFTWARE QUALITY:
* TESTS: No longer testing forced termination of forked cluster workers when
running on Solaris. The termination was done by launching a future that
called quit(), but that appeared to have corrupted the main R session when
running on Solaris.
DEPRECATED AND DEFUNCT:
* Formally defunct 'eager' and 'lazy' futures; use 'sequential' instead.
* Dropped previously defunct %<=% and %=>% operators.
Version: 1.5.0 [2017-05-24]
SIGNIFICANT CHANGES:
* Multicore and multisession futures no longer reserve one core for the
main R process, which was done to lower the risk for producing a higher
CPU load than the number of cores available for the R session.
NEW FEATURES:
* makeClusterPSOCK() now defaults to use the Windows PuTTY software's SSH
client 'plink -ssh', if 'ssh' is not found.
* Argument 'homogeneous' of makeNodePSOCK(), a helper function of
makeClusterPSOCK(), will default to FALSE also if the hostname is a
fully qualified domain name (FQDN), that is, it "contains periods".
For instance, c('node1', 'node2.server.org') will use homogeneous = TRUE
for the first worker and homogeneous = FALSE for the second.
* makeClusterPSOCK() now asserts that each cluster node is functioning by
retrieving and recording the node's session information including the
process ID of the corresponding R process.
* Nested futures sets option 'mc.cores' to prevent spawning of recursive
parallel processes by mistake. Because 'mc.cores' controls _additional_
processes, it was previously set to zero. However, since some functions
such as mclapply() does not support that, it is now set to one instead.
DOCUMENTATION:
* Help on makeClusterPSOCK() gained more detailed descriptions on arguments
and what their defaults are.
DEPRECATED AND DEFUNCT:
* Formally deprecated eager futures; use sequential instead.
BUG FIXES:
* future_lapply() with multicore / multisession futures, would use a
suboptimal workload balancing where it split up the data in one chunk too
many. This is no longer a problem because of how argument 'workers' is
now defined for those type of futures (see note on top).
* future_lapply(), as well as lazy multicore and lazy sequential futures, did
not respect option 'future.globals.resolve', but was hardcoded to always
resolve globals (future.globals.resolve = TRUE).
* When globals larger than the allowed size (option 'future.globals.maxSize')
are detected an informative error message is generated. Previous version
introduced a bug causing the error to produce another error.
* Lazy sequential futures would produce an error when resolved if required
packages had been detached.
* print() would not display globals gathered for lazy sequential futures.
SOFTWARE QUALITY:
* Added package tests for globals part of formulas part of other globals,
e.g. purrr::map(x, ~ rnorm(.)), which requires globals (>= 0.10.0).
* Now package tests with parallel::makeCluster() not only test for
type = 'PSOCK' clusters but also 'FORK' (when supported).
* TESTS: Cleaned up test scripts such that the overall processing time for
the tests was roughly halved, while preserving the same test coverage.
Version: 1.4.0 [2017-03-12]
SIGNIFICANT CHANGES:
* The default for future_lapply() is now to _not_ generate RNG seeds
(future.seed = FALSE). If proper random number generation is needed,
use future.seed = TRUE. For more details, see help page.
NEW FEATURES:
* future() and future_lapply() gained argument 'packages' for explicitly
specifying packages to be attached when the futures are evaluated.
Note that the default throughout the future package is that all globals
and all required packages are automatically identified and gathered, so
in most cases those do not have to be specified manually.
* The default values for arguments 'connectTimeout' and 'timeout' of
makeNodePSOCK() can now be controlled via global options.
RANDOM NUMBER GENERATION:
* Now future_lapply() guarantees that the RNG state of the calling R process
after returning is updated compared to what it was before and in the exact
same way regardless of 'future.seed' (except FALSE), 'future.scheduling'
and future strategy used. This is done in order to guarantee that an R
script calling future_lapply() multiple times should be numerically
reproducible given the same initial seed.
* It is now possible to specify a pre-generated sequence of .Random.seed
seeds to be used for each FUN(x[i], ...) call in future_lapply(x, FUN, ...).
PERFORMANCE:
* future_lapply() scans global variables for non-resolved futures (to resolve
them) and calculate their total size once. Previously, each chunk
(a future) would redo this.
BUG FIXES:
* Now future_lapply(x, FUN, ...) identifies global objects among 'x', 'FUN'
and '...' recursively until no new globals are found. Previously, only
the first level of globals were scanned. This is mostly thanks to a bug
fix in globals 0.9.0.
* A future that used a global object 'x' of a class that overrides length()
would produce an error if length(x) reports more elements than what can
be subsetted.
* nbrOfWorkers() gave an error with plan(cluster, workers = cl) where
'cl' is a cluster object created by parallel::makeCluster() etc.
This prevented for instance future_lapply() to work with such setups.
* plan(cluster, workers = cl) where cl <- makeCluster(..., type = MPI")
would give an instant error due to an invalid internal assertion.
DEPRECATED AND DEFUNCT:
* Previously deprecated arguments 'maxCores' and 'cluster' are now defunct.
* Previously deprecated assignment operators %<=% and %=>% are now defunct.
* availableCores(method = "mc.cores") is now defunct in favor of "mc.cores+1".
Version: 1.3.0 [2017-01-18]
SIGNIFICANT CHANGES:
* Where applicable, workers are now initiated when calling plan(), e.g.
plan(cluster) will set up workers on all cluster nodes. Previously,
this only happened when the first future was created.
NEW FEATURES:
* Renamed 'eager' futures to 'sequential', e.g. plan(sequential).
The 'eager' futures will be deprecated in an upcoming release.
* Added support for controlling whether a future is resolved eagerly
or lazily when creating the future, e.g. future(..., lazy = TRUE)
futureAssign(..., lazy = TRUE), and x %<-% { ... } %lazy% TRUE.
* future(), futureAssign() and futureCall() gained argument 'seed',
which specifies a L'Ecuyer-CMRG random seed to be used by the future.
The seed for future assignment can be specified via %seed%.
* futureAssign() now passes all additional arguments to future().
* Added future_lapply() which supports load balancing ("chunking")
and perfect reproducibility (regardless of type of load balancing
and how futures are resolved) via initial random seed.
* Added availableWorkers(). By default it returns localhost workers
according to availableCores(). In addition, it detects common
HPC allocations given in environment variables set by the HPC
scheduler.
* The default for plan(cluster) is now workers = availableWorkers().
* Now plan() stops any clusters that were implicitly created. For
instance, a multisession cluster created by plan(multisession) will
be stopped when plan(eager) is called.
* makeClusterPSOCK() treats workers that refer to a local machine by
its local or canonical hostname as "localhost". This avoids having
to launch such workers over SSH, which may not be supported on all
systems / compute cluster.
* Option 'future.debug' = TRUE also reports on total size of
globals identified and for cluster futures also the size of the
individual global variables exported.
* Option 'future.wait.timeout' (replaces 'future.wait.times') specifies
the maximum waiting time for a free workers (e.g. a core or a
compute node) before generating a timeout error.
* Option 'future.availableCores.fallback', which defaults to environment
variable 'R_FUTURE_AVAILABLECORES_FALLBACK' can now be used to specify
the default number of cores / workers returned by availableCores()
and availableWorkers() when no other settings are available. For
instance, if R_FUTURE_AVAILABLECORES_FALLBACK=1 is set system wide
in an HPC environment, then all R processes that uses availableCores()
to detect how many cores can be used will run as single-core processes.
Without this fallback setting, and without other core-specifying settings,
the default will be to use all cores on the machine, which does not play
well on multi-user systems.
GLOBALS:
* Globals part of locally defined functions are now also identified
thanks to globals (>= 0.8.0) updates.
DEPRECATED AND DEFUNCT:
* Lazy futures and plan(lazy) are now deprecated. Instead, use plan(eager)
and then f <- future(..., lazy = TRUE) or x %<-% { ... } %lazy% TRUE.
The reason behind this is that in some cases code that uses futures
only works under eager evaluation (lazy = FALSE; the default), or vice
verse. By removing the "lazy" future strategy, the user can no longer
override the lazy = TRUE / FALSE that the developer is using.
BUG FIXES:
* Creation of cluster futures (including multisession ones) would
time out already after 40 seconds if all workers were busy.
New default timeout is 30 days (option 'future.wait.timeout').
* nbrOfWorkers() gave an error for plan(cluster, workers) where 'workers'
was a character vector or a 'cluster' object of the parallel package.
Because of this, future_lapply() gave an error with such setups.
* availableCores(methods = "_R_CHECK_LIMIT_CORES_") would give an error
if not running R CMD check.
Version: 1.2.0 [2016-11-12]
NEW FEATURES: