Changelog
2021.03.0
Released on March 5, 2021
Note
This is the first release with support for Python 3.9 and the last release with support for Python 3.6
tcp.write()
: castmemoryview
to byte itemsize (:pr:`4555`) Mads R. B. Kristensen- Refcount the
thread_state.asynchronous
flag (:pr:`4557`) Mads R. B. Kristensen - Python 3.9 (:pr:`4460`) crusaderky
- Better bokeh defaults for dashboard (:pr:`4554`) Benjamin Zaitlen
- Expose system monitor dashboard as individual plot for lab extension (:pr:`4540`) Jacob Tomlinson
- Pass on original temp dir from nanny to worker (:pr:`4549`) Martin Durant
- Serialize and split (:pr:`4541`) Mads R. B. Kristensen
- Use the new HLG pack/unpack API in Dask (:pr:`4489`) Mads R. B. Kristensen
- Handle annotations for culled tasks (:pr:`4544`) Tom Augspurger
- Make sphinx autosummary and autoclass consistent (:pr:`4367`) Casey Clements
- Move
_transition*
toSchedulerState
(:pr:`4545`) jakirkham - Migrate from travis to GitHub actions (:pr:`4504`) crusaderky
- Move
new_task
toSchedulerState
(:pr:`4527`) jakirkham - Batch more Scheduler sends (:pr:`4526`) jakirkham
transition_memory_released
andget_nbytes()
optimizations (:pr:`4516`) jakirkham- Pin
black
pre-commit (:pr:`4533`) James Bourbeau - Read & write all frames in one pass (:pr:`4506`) jakirkham
- Skip
stream.write
call for empty frames (:pr:`4507`) jakirkham - Prepend frame metadata header (:pr:`4505`) jakirkham
transition_processing_memory
optimizations, etc. (:pr:`4487`) jakirkham- Attempt to get client from worker in
Queue
andVariable
(:pr:`4490`) James Bourbeau - Use
main
branch forzict
(:pr:`4499`) jakirkham - Use a callback to close TCP Comms, rather than check every time (:pr:`4453`) Matthew Rocklin
2021.02.0
Released on February 5, 2021
- Bump minimum Dask to 2021.02.0 (:pr:`4486`) James Bourbeau
- Update
TaskState
documentation about dependents attribute (:pr:`4440`) Florian Jetter - DOC: Autoreformat all functions docstrings (:pr:`4475`) Matthias Bussonnier
- Use cached version of
is_coroutine_function
in stream handling to (:pr:`4481`) Ian Rose - Optimize
transitions
(:pr:`4451`) jakirkham - Create
PULL_REQUEST_TEMPLATE.md
(:pr:`4476`) Ray Bell - DOC: typo, directives ends with 2 colons
::
(:pr:`4472`) Matthias Bussonnier - DOC: Proper numpydoc syntax for
distributed/protocol/*.py
(:pr:`4473`) Matthias Bussonnier - Update
pytest.skip
usage intest_server_listen
(:pr:`4467`) James Bourbeau - Unify annotations (:pr:`4406`) Ian Rose
- Added worker resources from config (:pr:`4456`) Tom Augspurger
- Fix var name in worker validation func (:pr:`4457`) Gil Forsyth
- Refactor
task_groups
&task_prefixes
(:pr:`4452`) jakirkham - Use
parent._tasks
inheartbeat
(:pr:`4450`) jakirkham - Refactor
SchedulerState
fromScheduler
(:pr:`4365`) jakirkham
2021.01.1
Released on January 22, 2021
- Make system monitor interval configurable (:pr:`4447`) Matthew Rocklin
- Add
uvloop
config value (:pr:`4448`) Matthew Rocklin - Additional optimizations to stealing (:pr:`4445`) jakirkham
- Give clusters names (:pr:`4426`) Jacob Tomlinson
- Use worker comm pool in
Semaphore
(:pr:`4195`) Florian Jetter - Set
runspec
on all new tasks to avoid deadlocks (:pr:`4432`) Florian Jetter - Support
TaskState
objects in story methods (:pr:`4434`) Matthew Rocklin - Support missing event loop in
Client.asynchronous
(:pr:`4436`) Matthew Rocklin - Don't require network to inspect tests (:pr:`4433`) Matthew Rocklin
2021.01.0
Released on January 15, 2021
- Add time started to scheduler info (:pr:`4425`) Jacob Tomlinson
- Log adaptive error (:pr:`4422`) Jacob Tomlinson
- Xfail normalization tests (:pr:`4411`) Jacob Tomlinson
- Use
dumps_msgpack
andloads_msgpack
when packing high level graphs (:pr:`4409`) Mads R. B. Kristensen - Add
nprocs
auto option todask-worker
CLI (:pr:`4377`) Jacob Tomlinson - Type annotation of
_reevaluate_occupancy_worker
(:pr:`4398`) jakirkham - Type
TaskGroup
inactive_states
(:pr:`4408`) jakirkham - Fix
test_as_current_is_thread_local
(:pr:`4402`) jakirkham - Use
list
comprehensions to bindTaskGroup
type (:pr:`4401`) jakirkham - Make tests pass after 2028 (:pr:`4403`) Bernhard M. Wiedemann
- Fix compilation warnings,
decide_worker
now a C func, stealing improvements (:pr:`4375`) jakirkham - Drop custom
__eq__
fromStatus
(:pr:`4270`) jakirkham test_performance_report
: skip without bokeh (:pr:`4388`) Bruno PaganiNanny
now respects dask settings from ctx mgr (:pr:`4378`) Florian Jetter- Better task duration estimates for outliers (:pr:`4213`) selshowk
- Dask internal inherit config (:pr:`4364`) Jacob Tomlinson
- Provide
setup.py
option to profile Cython code (:pr:`4362`) jakirkham - Optimizations of
*State
andTask*
objects and stealing (:pr:`4358`) jakirkham - Cast
SortedDict
s todict
s in a few key places & other minor changes (:pr:`4355`) jakirkham - Use task annotation priorities for user-level priorities (:pr:`4354`) James Bourbeau
- Added docs to highlevelgraph pack/unpack (:pr:`4352`) Mads R. B. Kristensen
- Optimizations in notable functions used by transitions (:pr:`4351`) jakirkham
- Silence exception when releasing futures on process shutdown (:pr:`4309`) Benjamin Zaitlen
2020.12.0
Released on December 10, 2020
Highlights
- Switched to CalVer for versioning scheme.
- The scheduler can now receives Dask
HighLevelGraph
s instead of raw dictionary task graphs. This allows for a much more efficient communication of task graphs from the client to the scheduler. - Added support for using custom
Layer
-level annotations likepriority
,retries
, etc. with thedask.annotations
context manager. - Updated minimum supported version of Dask to 2020.12.0.
- Added many type annotations and updates to allow for gradually Cythonizing the scheduler.
All changes
- Some common optimizations across transitions (:pr:`4348`) jakirkham
- Drop serialize extension (:pr:`4344`) jakirkham
- Log duplciate workers in scheduler (:pr:`4338`) Matthew Rocklin
- Annotation of some comm related methods in the
Scheduler
(:pr:`4341`) jakirkham - Optimize
assert
invalidate_waiting
(:pr:`4342`) jakirkham - Optimize
decide_worker
(:pr:`4332`) jakirkham - Store occupancy in
_reevaluate_occupancy_worker
(:pr:`4337`) jakirkham - Handle
WorkerState
memory_limit
ofNone
(:pr:`4335`) jakirkham - Use
bint
to annotate boolean attributes (:pr:`4334`) jakirkham - Optionally use offload executor in worker (:pr:`4307`) Matthew Rocklin
- Optimize
send_task_to_worker
(:pr:`4331`) jakirkham - Optimize
valid_workers
(:pr:`4329`) jakirkham - Store occupancy in
transition_waiting_processing
(:pr:`4330`) jakirkham - Optimize
get_comm_cost
(:pr:`4328`) jakirkham - Use
.pop(...)
to removekey
(:pr:`4327`) jakirkham - Use
operator.attrgetter
onWorkerState.address
(:pr:`4324`) jakirkham - Annotate
Task*
objects for Cythonization (:pr:`4302`) jakirkham - Ensure
retire_workers
alwaysreturn
adict
(:pr:`4323`) jakirkham - Some Cython fixes for
WorkerState
(:pr:`4321`) jakirkham - Optimize
WorkerState.__eq__
(:pr:`4320`) jakirkham - Swap order of
TaskGroup
andTaskPrefix
(:pr:`4319`) jakirkham - Check traceback object can be unpickled (:pr:`4299`) jakirkham
- Move
TaskGroup
&TaskPrefix
before TaskState (:pr:`4318`) jakirkham - Remove empty
test_highgraph.py
file (:pr:`4313`) James Bourbeau - Ensure that
retire_workers
returns adict
(:pr:`4315`) Matthew Rocklin - Annotate
WorkerState
for Cythonization (:pr:`4294`) jakirkham - Close
comm
on low-level errors (:pr:`4239`) jochen-ott-by - Coerce new
TaskState.nbytes
value toint
(:pr:`4311`) jakirkham - Remove offload
try
/except
forthread_name_prefix
keyword (:pr:`4308`) James Bourbeau - Fix
pip
install issue on CI (:pr:`4310`) jakirkham - Transmit
Layer
annotations to scheduler (:pr:`4279`) Simon Perkins - Ignores any compiled files generated by Cython (:pr:`4301`) jakirkham
- Protect against missing key in
get_metrics
(:pr:`4300`) Matthew Rocklin - Provide option to build Distributed with Cython (:pr:`4292`) jakirkham
- Set
WorkerState.processing
w/dict
inclean
(:pr:`4295`) jakirkham - Annotate
ClientState
for Cythonization (:pr:`4290`) jakirkham - Annotate
check_idle_saturated
for Cythonization (:pr:`4289`) jakirkham - Avoid flicker in
TaskStream
with "Scheduler is empty" message (:pr:`4284`) Matthew Rocklin - Make
gather_dep
robust to missing tasks (:pr:`4285`) Matthew Rocklin - Annotate
extract_serialize
(for Cythonization) (:pr:`4283`) jakirkham - Move
nbytes
from Worker's state toTaskState
(:pr:`4274`) Gil Forsyth - Drop extra type check in
_extract_serialize
(:pr:`4281`) jakirkham - Move Status to top-level import (:pr:`4280`) Matthew Rocklin
- Add
__hash__
and__eq__
forTaskState
(:pr:`4278`) jakirkham - Add
__hash__
and__eq__
forClientState
(:pr:`4276`) jakirkham - Collect
report
'sclient_key``s in a ``list
(:pr:`4275`) jakirkham - Precompute
hash
forWorkerState
(:pr:`4271`) jakirkham - Use
Status
Enum
inremove_worker
(:pr:`4269`) jakirkham - Add aggregated topic logs and
log_event
method (:pr:`4230`) James Bourbeau - Find the set of workers instead of their frequency (:pr:`4267`) jakirkham
- Use
set.update
to include othercomms
(:pr:`4268`) jakirkham - Support string timeouts in
sync
(:pr:`4266`) James Bourbeau - Use
dask.utils.stringify()
instead ofdistributed.utils.tokey()
(:pr:`4255`) Mads R. B. Kristensen - Use
.items()
to walk through keys and values (:pr:`4261`) jakirkham - Simplify frame length packing in TCP write (:pr:`4257`) jakirkham
- Comm/tcp listener: do not pass comm with failed handshake to
comm_handler
(:pr:`4240`) jochen-ott-by - Fuse steps in
extract_serialize
(:pr:`4254`) jakirkham - Drop
test_sklearn
(:pr:`4253`) jakirkham - Document task priority tie breaking (:pr:`4252`) James Bourbeau
__dask_distributed_pack__()
: client argument (:pr:`4248`) Mads R. B. Kristensen- Configurable timeouts for
worker_client
andget_client
(:pr:`4146`) GeethanjaliEswaran - Add dask/distributed versions to
performance_report
(:pr:`4249`) Matthew Rocklin - Update miniconda GitHub action (:pr:`4250`) James Bourbeau
- UCX closing ignore error (:pr:`4236`) Mads R. B. Kristensen
- Redirect to
dask-worker
cli documentation (:pr:`4247`) Timost - Upload file worker plugin (:pr:`4238`) Ian Rose
- Create dependency
TaskState
as needed ingather_dep
(:pr:`4241`) Gil Forsyth - Instantiate plugin if needed in
register_worker_plugin
(:pr:`4198`) Julia Signell - Allow actors to call actors on the same worker (:pr:`4225`) Martin Durant
- Special case profile thread in leaked thread check (:pr:`4229`) James Bourbeau
- Use
intersection()
on a set instead ofdict_keys
inupdate_graph
(:pr:`4227`) Mads R. B. Kristensen - Communicate
HighLevelGraphs
directly to theScheduler
(:pr:`4140`) Mads R. B. Kristensen - Add
get_task_metadata
context manager (:pr:`4216`) James Bourbeau - Task state logs and data fix (:pr:`4206`) Gil Forsyth
- Send active task durations from worker to scheduler (:pr:`4192`) James Bourbeau
- Fix state check in
test_close_gracefully
(:pr:`4203`) Gil Forsyth - Avoid materializing layers in
Client.compute()
(:pr:`4196`) Mads R. B. Kristensen - Add
TaskState
metadata (:pr:`4191`) James Bourbeau - Fix regression in task stealing for already released keys (:pr:`4182`) Florian Jetter
- Fix
_graph_to_futures
bug for futures-based dependencies (:pr:`4178`) Richard J Zamora - High level graph
dumps
/loads
support (:pr:`4174`) Mads R. B. Kristensen - Implement pass HighLevelGraphs through
_graph_to_futures
(:pr:`4139`) Mads R. B. Kristensen - Support
async
preload click commands (:pr:`4170`) James Bourbeau dask-worker
cli memory limit option doc fix (:pr:`4172`) marwan116- Add
TaskState
toworker.py
(:pr:`4107`) Gil Forsyth - Increase robustness of
Semaphore.release
(:pr:`4151`) Lucas Rademaker - Skip batched comm test win / tornado5 (:pr:`4166`) Tom Augspurger
- Set Zict buffer target to maxsize when
memory_target_fraction
isFalse
(:pr:`4156`) Krishan Bhasin - Add
PipInstall
WorkerPlugin
(:pr:`3216`) Matthew Rocklin - Log
KilledWorker
events in the scheduler (:pr:`4157`) Matthew Rocklin - Fix
test_gpu_metrics
failure (:pr:`4154`) jakirkham
2.30.1 - 2020-11-03
- Pin
pytest-asyncio
version (:pr:`4212`) James Bourbeau - Replace
AsyncProcess
exit handler byweakref.finalize
(:pr:`4184`) Peter Andreas Entschev - Remove hard coded connect handshake timeouts (:pr:`4176`) Florian Jetter
2.30.0 - 2020-10-06
- Support
SubgraphCallable
instr_graph()
(:pr:`4148`) Mads R. B. Kristensen - Handle exceptions in
BatchedSend
(:pr:`4135`) Tom Augspurger - Fix for missing
:
in autosummary docs (:pr:`4143`) Gil Forsyth - Limit GPU metrics to visible devices only (:pr:`3810`) Jacob Tomlinson
2.29.0 - 2020-10-02
- Use
pandas.testing
(:pr:`4138`) jakirkham - Fix a few typos (:pr:`4131`) Pav A
- Return right away in
Cluster.close
if cluster is already closed (:pr:`4116`) Tom Rochette - Update async doc with example on
.compute()
vsclient.compute()
(:pr:`4137`) Benjamin Zaitlen - Correctly tear down
LoopRunner
inClient
(:pr:`4112`) Sergey Kozlov - Simplify
Client._graph_to_futures()
(:pr:`4127`) Mads R. B. Kristensen - Cleanup new exception traceback (:pr:`4125`) Krishan Bhasin
- Stop writing config files by default (:pr:`4123`) Matthew Rocklin
2.28.0 - 2020-09-25
- Fix SSL
connection_args
forprogressbar
connect (:pr:`4122`) jennalc
2.27.0 - 2020-09-18
- Fix registering a worker plugin with
name
arg (:pr:`4105`) Nick Evans - Support different
remote_python
paths on cluster nodes (:pr:`4085`) Abdulelah Bin Mahfoodh - Allow
RuntimeError
s when closing global clients (:pr:`4115`) Matthew Rocklin - Match
pre-commit
in dask (:pr:`4049`) Julia Signell - Update
super
usage (:pr:`4110`) Poruri Sai Rahul
2.26.0 - 2020-09-11
- Add logging for adaptive start and stop (:pr:`4101`) Matthew Rocklin
- Don't close a nannied worker if it hasn't yet started (:pr:`4093`) Matthew Rocklin
- Respect timeouts when closing clients synchronously (:pr:`4096`) Matthew Rocklin
- Log when downloading a preload script (:pr:`4094`) Matthew Rocklin
dask-worker --nprocs
accepts negative values (:pr:`4089`) Dror Speiser- Support zero-worker clients (:pr:`4090`) Matthew Rocklin
- Exclude
fire-and-forget
client from metrics (:pr:`4078`) Tom Augspurger - Drop
Serialized.deserialize()
method (:pr:`4073`) jakirkham - Add
timeout=
keyword toClient.wait_for_workers
method (:pr:`4087`) Matthew Rocklin
2.25.0 - 2020-08-28
- Update for black (:pr:`4081`) Tom Augspurger
- Provide informative error when connecting an older version of Dask (:pr:`4076`) Matthew Rocklin
- Simplify
pack_frames
(:pr:`4068`) jakirkham - Simplify
frame_split_size
(:pr:`4067`) jakirkham - Use
list.insert
to add prelude up front (:pr:`4066`) jakirkham - Graph helper text (:pr:`4064`) Julia Signell
- Graph dashboard: Reset container data if task number is too large (:pr:`4056`) Florian Jetter
- Ensure semaphore picks correct
IOLoop
for threadpool workers (:pr:`4060`) Florian Jetter - Add cluster log method (:pr:`4051`) Jacob Tomlinson
- Cleanup more exception tracebacks (:pr:`4054`) Krishan Bhasin
- Improve documentation of
scheduler.locks
options (:pr:`4062`) Florian Jetter
2.24.0 - 2020-08-22
- Move toolbar to above and fix y axis (#4043) Julia Signell
- Make behavior clearer for how to get worker dashboard (#4047) Julia Signell
- Worker dashboard clean up (#4046) Julia Signell
- Add a default argument to the datasets and a possibility to override datasets (#4052) Nils Braun
- Discover HTTP endpoints (#3744) Martin Durant
2.23.0 - 2020-08-14
- Tidy up exception traceback in TCP Comms (:pr:`4042`) Krishan Bhasin
- Angle on the x-axis labels (:pr:`4030`) Mathieu Dugré
- Always set RMM's strides in the
header
(:pr:`4039`) jakirkham - Fix documentation
upload_file
(:pr:`4038`) Roberto Panai - Update UCX tests for new handshake step (:pr:`4036`) jakirkham
- Add test for informative errors in serialization cases (:pr:`4029`) Matthew Rocklin
- Add compression, pickle protocol to comm contexts (:pr:`4019`) Matthew Rocklin
- Make GPU plots robust to not having GPUs (:pr:`4008`) Matthew Rocklin
- Update
PendingDeprecationWarning
with correct version number (:pr:`4025`) Matthias Bussonnier - Install PyTorch on CI (:pr:`4017`) jakirkham
- Try getting cluster
dashboard_link
before asking scheduler (:pr:`4018`) Matthew Rocklin - Ignore writeable frames with builtin
array
(:pr:`4016`) jakirkham - Just extend
frames2
byframes
(:pr:`4015`) jakirkham - Serialize builtin array (:pr:`4013`) jakirkham
- Use cuDF's
assert_eq
(:pr:`4014`) jakirkham - Clear function cache whenever we upload a new file (:pr:`3993`) Jack Xiaosong Xu
- Emmit warning when assign/comparing string with
Status
Enum
(:pr:`3875`) Matthias Bussonnier - Track mutable frames (:pr:`4004`) jakirkham
- Improve
bytes
andbytearray
serialization (:pr:`4009`) jakirkham - Fix memory histogram values in dashboard (:pr:`4006`) Willi Rath
2.22.0 - 2020-07-31
- Only call
frame_split_size
when there are frames (:pr:`3996`) jakirkham - Fix failing
test_bandwidth
(:pr:`3999`) jakirkham - Handle sum of memory percentage when
memory_limit
is 0 (:pr:`3984`) Julia Signell - Drop msgpack pre-0.5.2 compat code (:pr:`3977`) jakirkham
- Revert to localhost for local IP if no network available (:pr:`3991`) Matthew Rocklin
- Add missing backtick in inline directive. (:pr:`3988`) Matthias Bussonnier
- Warn when
threads_per_worker
is set to zero (:pr:`3986`) Julia Signell - Use
memoryview
inunpack_frames
(:pr:`3980`) jakirkham - Iterate over list of comms (:pr:`3959`) Matthew Rocklin
- Streamline
pack_frames
/unpack_frames
frames (:pr:`3973`) jakirkham - Always attempt to create
dask-worker-space
folder and continue if it exists (:pr:`3972`) Jendrik Jördening - Use
merge_frames
with host memory only (:pr:`3971`) jakirkham - Simplify
pack_frames_prelude
(:pr:`3961`) jakirkham - Use continuation prompt for proper example parsing (:pr:`3966`) Matthias Bussonnier
- Ensure writable frames (:pr:`3967`) jakirkham
2.21.0 - 2020-07-17
- Fix data replication error (:pr:`3963`) Andrew Fulton
- Treat falsey local directory as
None
(:pr:`3964`) Tom Augspurger - Unpin
numpydoc
now that 1.1 is released (:pr:`3957`) Gil Forsyth - Error hard when Dask has mismatched versions or lz4 installed (:pr:`3936`) Matthew Rocklin
- Skip coercing to
bytes
inmerge_frames
(:pr:`3960`) jakirkham - UCX: reuse endpoints in order to fix NVLINK issue (:pr:`3953`) Mads R. B. Kristensen
- Optionally use
pickle5
(:pr:`3849`) jakirkham - Update time per task chart with filtering and pie (:pr:`3933`) Benjamin Zaitlen
- UCX: explicit shutdown message (:pr:`3950`) Mads R. B. Kristensen
- Avoid too aggressive retry of connections (:pr:`3944`) Matthias Bussonnier
- Parse timeouts in
Client.sync
(:pr:`3952`) Matthew Rocklin - Synchronize on non-trivial CUDA frame transmission (:pr:`3949`) jakirkham
- Serialize
memoryview
withshape
andformat
(:pr:`3947`) jakirkham - Move
scheduler_comm
intoCluster.__init__
(:pr:`3945`) Matthew Rocklin
2.20.0 - 2020-07-02
- Link issue on using
async
withexecutor_submit
(:pr:`3939`) jakirkham - Make dashboard server listens on all IPs by default even when interface is set explicitly (:pr:`3941`) Loïc Estève
- Update logic for worker removal in check ttl (:pr:`3927`) Benjamin Zaitlen
- Close a created cluster quietly (:pr:`3935`) Matthew Rocklin
- Ensure
Worker.run*
handleskwargs
correctly (:pr:`3937`) jakirkham - Restore
Scheduler.time_started
for Dask Gateway (:pr:`3934`) Tom Augspurger - Fix exception handling in
_wait_until_connected
(:pr:`3912`) Alexander Clausen - Make local directory if it does not exist (:pr:`3928`) Matthew Rocklin
- Install vanilla status route if bokeh dependency is not satisfied (:pr:`3844`) joshreback
- Make
Worker.delete_data
sync (:pr:`3922`) Peter Andreas Entschev - Fix
ensure_bytes
import location (:pr:`3919`) jakirkham - Fix race condition in repeated calls to
cluster.adapt()
(:pr:`3915`) Jacob Tomlinson
2.19.0 - 2020-06-19
- Notify worker plugins when a task is released (:pr:`3817`) Nick Evans
- Update heartbeat checks in scheduler (:pr:`3896`) Benjamin Zaitlen
- Make encryption default if
Security
is given arguments (:pr:`3887`) Matthew Rocklin - Show
cpu_fraction
on hover for dashboard workers circle plot. (:pr:`3906`) Loïc Estève - Prune virtual client on variable deletion (:pr:`3910`) Marco Neumann
- Fix total aggregated metrics in dashboard (:pr:`3897`) Loïc Estève
- Support Bokeh 2.1 (:pr:`3904`) Matthew Rocklin
- Update
related-work.rst
(:pr:`3889`) DomHudson - Skip
test_pid_file
in older versions of Python (:pr:`3888`) Matthew Rocklin - Replace
stream=
withcomm=
in handlers (:pr:`3860`) Julien Jerphanion - Check hosts for
None
value in SSH cluster. (:pr:`3883`) Matthias Bussonnier - Allow dictionaries in
security=
keywords (:pr:`3874`) Matthew Rocklin - Use pickle protocol 5 with NumPy object arrays (:pr:`3871`) jakirkham
- Cast any
frame
touint8
(same type asbytes
) (:pr:`3870`) jakirkham - Use
Enum
for worker, scheduler and nanny status. (:pr:`3853`) Matthias Bussonnier - Drop legacy
buffer_interface
assignment (:pr:`3869`) jakirkham - Drop old frame splitting in NumPy serialization (:pr:`3868`) jakirkham
- Drop no longer needed local
import pickle
(:pr:`3865`) jakirkham - Fix typo in
feed
's log message (:pr:`3867`) jakirkham - Tidy pickle (:pr:`3866`) jakirkham
- Handle empty times in task stream (:pr:`3862`) Benjamin Zaitlen
- Change
asyncssh
objects to sphinx references (:pr:`3861`) Jacob Tomlinson - Improve
SSHCluster
docstring forconnect_options
(:pr:`3859`) Jacob Tomlinson - Validate address parameter in client constructor (:pr:`3842`) joshreback
- Use
SpecCluster
name in worker names (:pr:`3855`) Loïc Estève - Allow async
add_worker
andremove_worker
plugin methods (:pr:`3847`) James Bourbeau
2.18.0 - 2020-06-05
- Merge frames in
deserialize_bytes
(:pr:`3639`) John Kirkham - Allow
SSHCluster
to take a list ofconnect_options
(:pr:`3854`) Jacob Tomlinson - Add favicon to performance report (:pr:`3852`) Jacob Tomlinson
- Add dashboard plots for the amount of time spent per key and for transfer/serialization (:pr:`3792`) Benjamin Zaitlen
- Fix variable name in journey of a task documentation (:pr:`3840`) Matthias Bussonnier
- Fix typo in journey of a task doc (:pr:`3838`) James Bourbeau
- Register
dask_cudf
serializers (:pr:`3832`) John Kirkham - Fix key check in
rebalance
missing keys (:pr:`3834`) Jacob Tomlinson - Allow collection of partial profile information in case of exceptions (:pr:`3773`) Florian Jetter
2.17.0 - 2020-05-26
- Record the time since the last run task on the scheduler (:pr:`3830`) Matthew Rocklin
- Set colour of
nbytes
pane based on thresholds (:pr:`3805`) Krishan Bhasin - Include total number of tasks in the performance report (:pr:`3822`) Abdulelah Bin Mahfoodh
- Allow to pass in task key strings in the worker restrictions (:pr:`3826`) Nils Braun
- Control de/ser offload (:pr:`3793`) Martin Durant
- Parse timeout parameters in
Variable
/Event
/Lock
to support text timeouts (:pr:`3825`) Nils Braun - Don't send empty dependencies (:pr:`3423`) Jakub Beránek
- Add distributed Dask
Event
that mimicsthreading.Event
(:pr:`3821`) Nils Braun - Enhance
VersionMismatchWarning
messages (:pr:`3786`) Abdulelah Bin Mahfoodh - Support Pickle's protocol 5 (:pr:`3784`) jakirkham
- Replace
utils.ignoring
withcontextlib.suppress
(:pr:`3819`) Nils Braun - Make re-creating conda environments from the CI output easier (:pr:`3816`) Lucas Rademaker
- Add prometheus metrics for semaphore (:pr:`3757`) Lucas Rademaker
- Fix worker plugin called with superseded transition (:pr:`3812`) Nick Evans
- Add retries to server listen (:pr:`3801`) Jacob Tomlinson
- Remove commented out lines from
scheduler.py
(:pr:`3803`) James Bourbeau - Fix
RuntimeWarning
for never awaited coroutine when usingdistributed.Semaphore
(:pr:`3713`) Florian Jetter - Fix profile thread leakage during test teardown on some platforms (:pr:`3795`) Florian Jetter
- Await self before handling comms (:pr:`3788`) Matthew Rocklin
- Fix typo in
Cluster
docstring (:pr:`3787`) Scott Sanderson
2.16.0 - 2020-05-08
Client.get_dataset
to always createFutures
attached to itself (:pr:`3729`) crusaderky- Remove dev-requirements since it is unused (:pr:`3782`) Julia Signell
- Use bokeh column for
/system
instead of custom css (:pr:`3781`) Julia Signell - Attempt to fix
test_preload_remote_module
on windows (:pr:`3775`) James Bourbeau - Fix broadcast for TLS comms (:pr:`3766`) Florian Jetter
- Don't validate http preloads locally (:pr:`3768`) Rami Chowdhury
- Allow range of ports to be specified for
Workers
(:pr:`3704`) James Bourbeau - Add UCX support for RDMACM (:pr:`3759`) Peter Andreas Entschev
- Support web addresses in preload (:pr:`3755`) Matthew Rocklin
2.15.2 - 2020-05-01
- Connect to dashboard when address provided (:pr:`3758`) Tom Augspurger
- Move
test_gpu_metrics test
(:pr:`3721`) Tom Augspurger - Nanny closing worker on
KeyboardInterrupt
(:pr:`3747`) Mads R. B. Kristensen - Replace
OrderedDict
withdict
in scheduler (:pr:`3740`) Matthew Rocklin - Fix exception handling typo (:pr:`3751`) Jonas Haag
2.15.1 - 2020-04-28
- Ensure
BokehTornado
uses prefix (:pr:`3746`) James Bourbeau - Warn if cluster closes before starting (:pr:`3735`) Matthew Rocklin
- Memoryview serialisation (:pr:`3743`) Martin Durant
- Allows logging config under distributed key (:pr:`2952`) Dillon Niederhut
2.15.0 - 2020-04-24
- Reinstate support for legacy
@gen_cluster
functions (:pr:`3738`) crusaderky - Relax NumPy requirement in UCX (:pr:`3731`) jakirkham
- Add Configuration Schema (:pr:`3696`) Matthew Rocklin
- Reuse CI scripts for local installation process (:pr:`3698`) crusaderky
- Use
PeriodicCallback
class from tornado (:pr:`3725`) James Bourbeau - Add
remote_python
option in ssh cmd (:pr:`3709`) Abdulelah Bin Mahfoodh - Configurable polling interval for cluster widget (:pr:`3723`) Julia Signell
- Fix copy-paste in docs (:pr:`3728`) Julia Signell
- Replace
gen.coroutine
with async-await in tests (:pr:`3706`) crusaderky - Fix flaky
test_oversubscribing_leases
(:pr:`3726`) Florian Jetter - Add
batch_size
toClient.map
(:pr:`3650`) Tom Augspurger - Adjust semaphore test timeouts (:pr:`3720`) Florian Jetter
- Dask-serialize dicts longer than five elements (:pr:`3689`) Richard J Zamora
- Force
threads_per_worker
(:pr:`3715`) crusaderky - Idempotent semaphore acquire with retries (:pr:`3690`) Florian Jetter
- Always use
readinto
in TCP (:pr:`3711`) jakirkham - Avoid
DeprecationWarning
from pandas (:pr:`3712`) Tom Augspurger - Allow modification of
distributed.comm.retry
at runtime (:pr:`3705`) Florian Jetter - Do not log an error on unset variable delete (:pr:`3652`) Jonathan J. Helmus
- Add
remote_python
keyword to the newSSHCluster
(:pr:`3701`) Abdulelah Bin Mahfoodh - Replace Example with Examples in docstrings (:pr:`3697`) Matthew Rocklin
- Add
Cluster
__enter__
and__exit__
methods (:pr:`3699`) Matthew Rocklin - Fix propagating inherit config in
SSHCluster
for non-bash shells (:pr:`3688`) Abdulelah Bin Mahfoodh - Add
Client.wait_to_workers
toClient
autosummary table (:pr:`3692`) James Bourbeau - Replace Bokeh Server with Tornado HTTPServer (:pr:`3658`) Matthew Rocklin
- Fix
dask-ssh
after removinglocal-directory
fromdask_scheduler
cli (:pr:`3684`) Abdulelah Bin Mahfoodh - Support preload modules in
Nanny
(:pr:`3678`) Matthew Rocklin - Refactor semaphore internals: make
_get_lease
synchronous (:pr:`3679`) Lucas Rademaker - Don't make task graphs too big (:pr:`3671`) Martin Durant
- Pass through
connection
/listen_args
as splatted keywords (:pr:`3674`) Matthew Rocklin - Run preload at import, start, and teardown (:pr:`3673`) Matthew Rocklin
- Use relative URL in scheduler dashboard (:pr:`3676`) Nicholas Smith
- Expose
Security
object as public API (:pr:`3675`) Matthew Rocklin - Add zoom tools to profile plots (:pr:`3672`) James Bourbeau
- Update
Scheduler.rebalance
return value when data is missing (:pr:`3670`) James Bourbeau
2.14.0 - 2020-04-03
- Enable more UCX tests (:pr:`3667`) jakirkham
- Remove openssl 1.1.1d pin for Travis (:pr:`3668`) Jonathan J. Helmus
- More documentation for
Semaphore
(:pr:`3664`) Florian Jetter - Get CUDA context to finalize Numba
DeviceNDArray
(:pr:`3666`) jakirkham - Add Resouces option to
get_task_stream
and calloutput_file
(:pr:`3653`) Prasun Anand - Add
Semaphore
extension (:pr:`3573`) Lucas Rademaker - Replace
ncores
withnthreads
in work stealing tests (:pr:`3615`) James Bourbeau - Clean up some test warnings (:pr:`3662`) Matthew Rocklin
- Write "why killed" docs (:pr:`3596`) Martin Durant
- Update Python version checking (:pr:`3660`) James Bourbeau
- Add newlines to ensure code formatting for
retire_workers
(:pr:`3661`) Rami Chowdhury - Clean up performance report test (:pr:`3655`) Matthew Rocklin
- Avoid diagnostics time in performance report (:pr:`3654`) Matthew Rocklin
- Introduce config for default task duration (:pr:`3642`) Gabriel Sailer
- UCX simplify receiving frames in
comm
(:pr:`3651`) jakirkham - Bump checkout GitHub action to v2 (:pr:`3649`) James Bourbeau
- Handle exception in
faulthandler
(:pr:`3646`) Jacob Tomlinson - Add prometheus metric for suspicious tasks (:pr:`3550`) Gabriel Sailer
- Remove
local-directory
keyword (:pr:`3620`) Prasun Anand - Don't create output Futures in Client when there are mixed Client Futures (:pr:`3643`) James Bourbeau
- Add link to
contributing.md
(:pr:`3621`) Prasun Anand - Update bokeh dependency in CI builds (:pr:`3637`) James Bourbeau
2.13.0 - 2020-03-25
- UCX synchronize default stream only on CUDA frames (:pr:`3638`) Peter Andreas Entschev
- Add
as_completed.clear
method (:pr:`3617`) Matthew Rocklin - Drop unused line from
pack_frames_prelude
(:pr:`3634`) John Kirkham - Add logging message when closing idle dask scheduler (:pr:`3632`) Matthew Rocklin
- Include frame lengths of CUDA objects in
header["lengths"]
(:pr:`3631`) John Kirkham - Ensure
Client
connection pool semaphore attaches to theClient
event loop (:pr:`3546`) James Bourbeau - Remove dead stealing code (:pr:`3619`) Florian Jetter
- Check
nbytes
andtypes
before readingdata
(:pr:`3628`) John Kirkham - Ensure that we don't steal blacklisted fast tasks (:pr:`3591`) Florian Jetter
- Support async
Listener.stop
functions (:pr:`3613`) Matthew Rocklin - Add str/repr methods to
as_completed
(:pr:`3618`) Matthew Rocklin - Add backoff to comm connect attempts. (:pr:`3496`) Matthias Urlichs
- Make
Listeners
awaitable (:pr:`3611`) Matthew Rocklin - Increase number of visible mantissas in dashboard plots (:pr:`3585`) Scott Sievert
- Pin openssl to 1.1.1d for Travis (:pr:`3602`) Jacob Tomlinson
- Replace
tornado.queues
withasyncio.queues
(:pr:`3607`) James Bourbeau - Remove
dill
from CI environments (:pr:`3608`) Loïc Estève - Fix linting errors (:pr:`3604`) James Bourbeau
- Synchronize default CUDA stream before UCX send/recv (:pr:`3598`) Peter Andreas Entschev
- Add configuration for
Adaptive
arguments (:pr:`3509`) Gabriel Sailer - Change
Adaptive
docs to referenceadaptive_target
(:pr:`3597`) Julia Signell - Optionally compress on a frame-by-frame basis (:pr:`3586`) Matthew Rocklin
- Add Python version to version check (:pr:`3567`) James Bourbeau
- Import
tlz
(:pr:`3579`) John Kirkham - Pin
numpydoc
to avoid double escaped*
(:pr:`3530`) Gil Forsyth - Avoid
performance_report
crashing when a worker dies mid-compute (:pr:`3575`) Krishan Bhasin - Pin
bokeh
in CI builds (:pr:`3570`) James Bourbeau - Disable fast fail on GitHub Actions Windows CI (:pr:`3569`) James Bourbeau
- Fix typo in
Client.shutdown
docstring (:pr:`3562`) John Kirkham - Add
local_directory
option todask-ssh
(:pr:`3554`) Abdulelah Bin Mahfoodh
2.12.0 - 2020-03-06
- Update
TaskGroup
remove logic (:pr:`3557`) James Bourbeau - Fix-up CuPy sparse serialization (:pr:`3556`) John Kirkham
- API docs for
LocalCluster
andSpecCluster
(:pr:`3548`) Tom Augspurger - Serialize sparse arrays (:pr:`3545`) John Kirkham
- Allow tasks with restrictions to be stolen (:pr:`3069`) Stan Seibert
- Use UCX default configuration instead of raising (:pr:`3544`) Peter Andreas Entschev
- Support using other serializers with
register_generic
(:pr:`3536`) John Kirkham - DOC: update to async await (:pr:`3543`) Tom Augspurger
- Use
pytest.raises
intest_ucx_config.py
(:pr:`3541`) John Kirkham - Fix/more ucx config options (:pr:`3539`) Benjamin Zaitlen
- Update heartbeat
CommClosedError
error handling (:pr:`3529`) James Bourbeau - Use
makedirs
when constructinglocal_directory
(:pr:`3538`) John Kirkham - Mark
None
as MessagePack serializable (:pr:`3537`) John Kirkham - Mark
bool
as MessagePack serializable (:pr:`3535`) John Kirkham - Use 'temporary-directory' from
dask.config
for Nanny's directory (:pr:`3531`) John Kirkham - Add try-except around getting source code in performance report (:pr:`3505`) Matthew Rocklin
- Fix typo in docstring (:pr:`3528`) Davis Bennett
- Make work stealing callback time configurable (:pr:`3523`) Lucas Rademaker
- RMM/UCX Config Flags (:pr:`3515`) Benjamin Zaitlen
- Revise develop-docs: conda env example (:pr:`3406`) Darren Weber
- Remove
import ucp
from the top ofucx.py
(:pr:`3510`) Peter Andreas Entschev - Rename
logs
toget_logs
(:pr:`3473`) Jacob Tomlinson - Stop keep alives when worker reconnecting to the scheduler (:pr:`3493`) Jacob Tomlinson
2.11.0 - 2020-02-19
- Add dask serialization of CUDA objects (:pr:`3482`) John Kirkham
- Suppress cuML
ImportError
(:pr:`3499`) John Kirkham - Msgpack 1.0 compatibility (:pr:`3494`) James Bourbeau
- Register cuML serializers (:pr:`3485`) John Kirkham
- Check exact equality for worker state (:pr:`3483`) Brett Naul
- Serialize 1-D, contiguous,
uint8
CUDA frames (:pr:`3475`) John Kirkham - Update NumPy array serialization to handle non-contiguous slices (:pr:`3474`) James Bourbeau
- Propose fix for collection based resources docs (:pr:`3480`) Chris Roat
- Remove
--verbose
flag from CI runs (:pr:`3484`) Matthew Rocklin - Do not duplicate messages in scheduler report (:pr:`3477`) Jakub Beránek
- Register Dask cuDF serializers (:pr:`3478`) John Kirkham
- Add support for Python 3.8 (:pr:`3249`) James Bourbeau
- Add last seen column to worker table and highlight errant workers (:pr:`3468`) kaelgreco
- Change default value of
local_directory
from empty string toNone
(:pr:`3441`) condoratberlin - Clear old docs (:pr:`3458`) Matthew Rocklin
- Change default multiprocessing behavior to spawn (:pr:`3461`) Matthew Rocklin
- Split dashboard host on additional slashes to handle inproc (:pr:`3466`) Jacob Tomlinson
- Update
locality.rst
(:pr:`3470`) Dustin Tindall - Minor
gen.Return
cleanup (:pr:`3469`) James Bourbeau - Update comparison logic for worker state (:pr:`3321`) rockwellw
- Update minimum
tblib
version to 1.6.0 (:pr:`3451`) James Bourbeau - Add total row to workers plot in dashboard (:pr:`3464`) Julia Signell
- Workaround
RecursionError
on profile data (:pr:`3455`) Tom Augspurger - Include code and summary in performance report (:pr:`3462`) Matthew Rocklin
- Skip
test_open_close_many_workers
on Python 3.6 (:pr:`3459`) Matthew Rocklin - Support serializing/deserializing
rmm.DeviceBuffer
s (:pr:`3442`) John Kirkham - Always add new
TaskGroup
toTaskPrefix
(:pr:`3322`) James Bourbeau - Rerun
black
on the code base (:pr:`3444`) John Kirkham - Ensure
__causes__
s of exceptions raised on workers are serialized (:pr:`3430`) Alex Adamson - Adjust
numba.cuda
import and add check (:pr:`3446`) John Kirkham - Fix name of Numba serialization test (:pr:`3447`) John Kirkham
- Checks for command parameters in
ssh2
(:pr:`3078`) Peter Andreas Entschev - Update
worker_kwargs
description inLocalCluster
constructor (:pr:`3438`) James Bourbeau - Ensure scheduler updates task and worker states after successful worker data deletion (:pr:`3401`) James Bourbeau
- Avoid
loop=
keyword in asyncio coordination primitives (:pr:`3437`) Matthew Rocklin - Call pip as a module to avoid warnings (:pr:`3436`) Cyril Shcherbin
- Add documentation of parameters in coordination primitives (:pr:`3434`) Søren Fuglede Jørgensen
- Replace
tornado.locks
with asyncio for Events/Locks/Conditions/Semaphore (:pr:`3397`) Matthew Rocklin - Remove object from class hierarchy (:pr:`3432`) Anderson Banihirwe
- Add
dashboard_link
property toClient
(:pr:`3429`) Jacob Tomlinson - Allow memory monitor to evict data more aggressively (:pr:`3424`) fjetter
- Make
_get_ip
return an IP address when defaulting (:pr:`3418`) Pierre Glaser - Support version checking with older versions of Dask (:pr:`3390`) Igor Gotlibovych
- Add Mac OS build to CI (:pr:`3358`) James Bourbeau
2.10.0 - 2020-01-28
- Fixed
ZeroDivisionError
in dashboard when no workers were present (:pr:`3407`) James Bourbeau - Respect the
dashboard-prefix
when redirecting from the root (:pr:`3387`) Chrysostomos Nanakos - Allow enabling / disabling work-stealing after the cluster has started (:pr:`3410`) John Kirkham
- Support
*args
and**kwargs
in offload (:pr:`3392`) Matthew Rocklin - Add lifecycle hooks to SchedulerPlugin (:pr:`3391`) Matthew Rocklin
2.9.3 - 2020-01-17
- Raise
RuntimeError
if no running loop (:pr:`3385`) James Bourbeau - Fix
get_running_loop
import (:pr:`3383`) James Bourbeau - Get JavaScript document location instead of window and handle proxied url (:pr:`3382`) Jacob Tomlinson
2.9.2 - 2020-01-16
- Move Windows CI to GitHub Actions (:pr:`3373`) Jacob Tomlinson
- Add client join and leave hooks (:pr:`3371`) Jacob Tomlinson
- Add cluster map dashboard (:pr:`3361`) Jacob Tomlinson
- Close connection comm on retry (:pr:`3365`) James Bourbeau
- Fix scheduler state in case of worker name collision (:pr:`3366`) byjott
- Add
--worker-class
option todask-worker
CLI (:pr:`3364`) James Bourbeau - Remove
locale
check that fails on OS X (:pr:`3360`) Jacob Tomlinson - Rework version checking (:pr:`2627`) Matthew Rocklin
- Add websocket scheduler plugin (:pr:`3335`) Jacob Tomlinson
- Return task in
dask-worker
on_signal
function (:pr:`3354`) James Bourbeau - Fix failures on mixed integer/string worker names (:pr:`3352`) Benedikt Reinartz
- Avoid calling
nbytes
multiple times when sending data (:pr:`3349`) Markus Mohrhard - Avoid setting event loop policy if within IPython kernel and no running event loop (:pr:`3336`) Mana Borwornpadungkitti
- Relax intermittent failing
test_profile_server
(:pr:`3346`) Matthew Rocklin
2.9.1 - 2019-12-27
- Add lock around dumps_function cache (:pr:`3337`) Matthew Rocklin
- Add setuptools to dependencies (:pr:`3320`) James Bourbeau
- Use TaskPrefix.name in Graph layout (:pr:`3328`) Matthew Rocklin
- Add missing " in performance report example (:pr:`3329`) John Kirkham
- Add performance report docs and color definitions to docs (:pr:`3325`) Benjamin Zaitlen
- Switch startstops to dicts and add worker name to transfer (:pr:`3319`) Jacob Tomlinson
- Add plugin entry point for out-of-tree comms library (:pr:`3305`) Patrick Sodré
- All scheduler task states in prometheus (:pr:`3307`) fjetter
- Use worker name in logs (:pr:`3309`) Stephan Erb
- Add TaskGroup and TaskPrefix scheduler state (:pr:`3262`) Matthew Rocklin
- Update latencies with heartbeats (:pr:`3310`) fjetter
- Update inlining Futures in task graph in Client._graph_to_futures (:pr:`3303`) James Bourbeau
- Use hostname as default IP address rather than localhost (:pr:`3308`) Matthew Rocklin
- Clean up flaky test_nanny_throttle (:pr:`3295`) Tom Augspurger
- Add lock to scheduler for sensitive operations (:pr:`3259`) Matthew Rocklin
- Log address for each of the Scheduler listerners (:pr:`3306`) Matthew Rocklin
- Make ConnectionPool.close asynchronous (:pr:`3304`) Matthew Rocklin
2.9.0 - 2019-12-06
- Add
dask-spec
CLI tool (:pr:`3090`) Matthew Rocklin - Connectionpool: don't hand out closed connections (:pr:`3301`) byjott
- Retry operations on network issues (:pr:`3294`) byjott
- Skip
Security.temporary()
tests if cryptography not installed (:pr:`3302`) James Bourbeau - Support multiple listeners in the scheduler (:pr:`3288`) Matthew Rocklin
- Updates RMM comment to the correct release (:pr:`3299`) John Kirkham
- Add title to
performance_report
(:pr:`3298`) Matthew Rocklin - Forgot to fix slow test (:pr:`3297`) Benjamin Zaitlen
- Update
SSHCluster
docstring parameters (:pr:`3296`) James Bourbeau worker.close()
awaitsbatched_stream.close()
(:pr:`3291`) Mads R. B. Kristensen- Fix asynchronous listener in UCX (:pr:`3292`) Benjamin Zaitlen
- Avoid repeatedly adding deps to already in memory stack (:pr:`3293`) James Bourbeau
- xfail ucx empty object typed dataframe (:pr:`3279`) Benjamin Zaitlen
- Fix
distributed.wait
documentation (:pr:`3289`) Tom Rochette - Move Python 3 syntax tests into main tests (:pr:`3281`) Matthew Rocklin
- xfail
test_workspace_concurrency
for Python 3.6 (:pr:`3283`) Matthew Rocklin - Add
performance_report
context manager for static report generation (:pr:`3282`) Matthew Rocklin - Update function serialization caches with custom LRU class (:pr:`3260`) James Bourbeau
- Make
Listener.start
asynchronous (:pr:`3278`) Matthew Rocklin - Remove
dask-submit
anddask-remote
(:pr:`3280`) Matthew Rocklin - Worker profile server (:pr:`3274`) Matthew Rocklin
- Improve bandwidth workers plot (:pr:`3273`) Matthew Rocklin
- Make profile coroutines consistent between
Scheduler
andWorker
(:pr:`3277`) Matthew Rocklin - Enable saving profile information from server threads (:pr:`3271`) Matthew Rocklin
- Remove memory use plot (:pr:`3269`) Matthew Rocklin
- Add offload size to configuration (:pr:`3270`) Matthew Rocklin
- Fix layout scaling on profile plots (:pr:`3268`) Jacob Tomlinson
- Set
x_range
in CPU plot based on the number of threads (:pr:`3266`) Matthew Rocklin - Use base-2 values for byte-valued axes in dashboard (:pr:`3267`) Matthew Rocklin
- Robust gather in case of connection failures (:pr:`3246`) fjetter
- Use
DeviceBuffer
from newer RMM releases (:pr:`3261`) John Kirkham - Fix dev requirements for pytest (:pr:`3264`) Elliott Sales de Andrade
- Add validate options to configuration (:pr:`3258`) Matthew Rocklin
2.8.1 - 2019-11-22
- Fix hanging worker when the scheduler leaves (:pr:`3250`) Tom Augspurger
- Fix NumPy writeable serialization bug (:pr:`3253`) James Bourbeau
- Skip
numba.cuda
tests if CUDA is not available (:pr:`3255`) Peter Andreas Entschev - Add new dashboard plot for memory use by key (:pr:`3243`) Matthew Rocklin
- Fix
array.shape()
->array.shape
(:pr:`3247`) Jed Brown - Fixed typos in
pubsub.py
(:pr:`3244`) He Jia - Fixed cupy array going out of scope (:pr:`3240`) Mads R. B. Kristensen
- Remove
gen.coroutine
usage in scheduler (:pr:`3242`) Jim Crist-Harif - Use
inspect.isawaitable
where relevant (:pr:`3241`) Jim Crist-Harif
2.8.0 - 2019-11-14
- Add UCX config values (:pr:`3135`) Matthew Rocklin
- Relax test_MultiWorker (:pr:`3210`) Matthew Rocklin
- Avoid ucp.init at import time (:pr:`3211`) Matthew Rocklin
- Clean up rpc to avoid intermittent test failure (:pr:`3215`) Matthew Rocklin
- Respect protocol if given to Scheduler (:pr:`3212`) Matthew Rocklin
- Use legend_field= keyword in bokeh plots (:pr:`3218`) Matthew Rocklin
- Cache psutil.Process object in Nanny (:pr:`3207`) Matthew Rocklin
- Replace gen.sleep with asyncio.sleep (:pr:`3208`) Matthew Rocklin
- Avoid offloading serialization for small messages (:pr:`3224`) Matthew Rocklin
- Add desired_workers metric (:pr:`3221`) Gabriel Sailer
- Fail fast when importing distributed.comm.ucx (:pr:`3228`) Matthew Rocklin
- Add module name to Future repr (:pr:`3231`) Matthew Rocklin
- Add name to Pub/Sub repr (:pr:`3235`) Matthew Rocklin
- Import CPU_COUNT from dask.system (:pr:`3199`) James Bourbeau
- Efficiently serialize zero strided NumPy arrays (:pr:`3180`) James Bourbeau
- Cache function deserialization in workers (:pr:`3234`) Matthew Rocklin
- Respect ordering of futures in futures_of (:pr:`3236`) Matthew Rocklin
- Bump dask dependency to 2.7.0 (:pr:`3237`) James Bourbeau
- Avoid setting inf x_range (:pr:`3229`) rockwellw
- Clear task stream based on recent behavior (:pr:`3200`) Matthew Rocklin
- Use the percentage field for profile plots (:pr:`3238`) Matthew Rocklin
2.7.0 - 2019-11-08
This release drops support for Python 3.5
- Adds badges to README.rst [skip ci] (:pr:`3152`) James Bourbeau
- Don't overwrite self.address if it is present (:pr:`3153`) Gil Forsyth
- Remove outdated references to debug scheduler and worker bokeh pages. (:pr:`3160`) darindf
- Update CONTRIBUTING.md (:pr:`3159`) Jacob Tomlinson
- Add Prometheus metric for a worker's executing tasks count (:pr:`3163`) darindf
- Update Prometheus documentation (:pr:`3165`) darindf
- Fix Numba serialization when strides is None (:pr:`3166`) Peter Andreas Entschev
- Await cluster in Adaptive.recommendations (:pr:`3168`) Simon Boothroyd
- Support automatic TLS (:pr:`3164`) Jim Crist
- Avoid swamping high-memory workers with data requests (:pr:`3071`) Tom Augspurger
- Update UCX variables to use sockcm by default (:pr:`3177`) Peter Andreas Entschev
- Get protocol in Nanny/Worker from scheduler address (:pr:`3175`) Peter Andreas Entschev
- Add worker and tasks state for Prometheus data collection (:pr:`3174`) darindf
- Use async def functions for offload to/from_frames (:pr:`3171`) Mads R. B. Kristensen
- Subprocesses inherit the global dask config (:pr:`3192`) Mads R. B. Kristensen
- XFail test_open_close_many_workers (:pr:`3194`) Matthew Rocklin
- Drop Python 3.5 (:pr:`3179`) James Bourbeau
- UCX: avoid double init after fork (:pr:`3178`) Mads R. B. Kristensen
- Silence warning when importing while offline (:pr:`3203`) James A. Bednar
- Adds docs to Client methods for resources, actors, and traverse (:pr:`2851`) IPetrik
- Add test for concurrent scatter operations (:pr:`2244`) Matthew Rocklin
- Expand async docs (:pr:`2293`) Dave Hirschfeld
- Add PatchedDeviceArray to drop stride attribute for cupy<7.0 (:pr:`3198`) Richard J Zamora
2.6.0 - 2019-10-15
- Refactor dashboard module (:pr:`3138`) Jacob Tomlinson
- Use
setuptools.find_packages
insetup.py
(:pr:`3150`) Matthew Rocklin - Move death timeout logic up to
Node.start
(:pr:`3115`) Matthew Rocklin - Only include metric in
WorkerTable
if it is a scalar (:pr:`3140`) Matthew Rocklin - Add
Nanny(config={...})
keyword (:pr:`3134`) Matthew Rocklin - Xfail
test_worksapce_concurrency
on Python 3.6 (:pr:`3132`) Matthew Rocklin - Extend Worker plugin API with transition method (:pr:`2994`) matthieubulte
- Raise exception if the user passes in unused keywords to
Client
(:pr:`3117`) Jonathan De Troye - Move new
SSHCluster
to top level (:pr:`3128`) Matthew Rocklin - Bump dask dependency (:pr:`3124`) Jim Crist
2.5.2 - 2019-10-04
- Make dask-worker close quietly when given sigint signal (:pr:`3116`) Matthew Rocklin
- Replace use of tornado.gen with asyncio in dask-worker (:pr:`3114`) Matthew Rocklin
- UCX: allocate CUDA arrays using RMM and Numba (:pr:`3109`) Mads R. B. Kristensen
- Support calling cluster.scale as async method (:pr:`3110`) Jim Crist
- Identify lost workers in SpecCluster based on address not name (:pr:`3088`) James Bourbeau
- Add Client.shutdown method (:pr:`3106`) Matthew Rocklin
- Collect worker-worker and type bandwidth information (:pr:`3094`) Matthew Rocklin
- Send noise over the wire to keep dask-ssh connection alive (:pr:`3105`) Gil Forsyth
- Retry scheduler connect multiple times (:pr:`3104`) Jacob Tomlinson
- Add favicon of logo to the dashboard (:pr:`3095`) James Bourbeau
- Remove utils.py functions for their dask/utils.py equivalents (:pr:`3042`) Matthew Rocklin
- Lower default bokeh log level (:pr:`3087`) Philipp Rudiger
- Check if self.cluster.scheduler is a local scheduler (:pr:`3099`) Jacob Tomlinson
2.5.1 - 2019-09-27
- Support clusters that don't have .security or ._close methods (:pr:`3100`) Matthew Rocklin
2.5.0 - 2019-09-27
- Use the new UCX Python bindings (:pr:`3059`) Mads R. B. Kristensen
- Fix worker preload config (:pr:`3027`) byjott
- Fix widget with spec that generates multiple workers (:pr:`3067`) Loïc Estève
- Make Client.get_versions async friendly (:pr:`3064`) Jacob Tomlinson
- Add configuation option for longer error tracebacks (:pr:`3086`) Daniel Farrell
- Have Client get Security from passed Cluster (:pr:`3079`) Matthew Rocklin
- Respect Cluster.dashboard_link in Client._repr_html_ if it exists (:pr:`3077`) Matthew Rocklin
- Add monitoring with dask cluster docs (:pr:`3072`) Arpit Solanki
- Protocol of cupy and numba handles serialization exclusively (:pr:`3047`) Mads R. B. Kristensen
- Allow specification of worker type in SSHCLuster (:pr:`3061`) Jacob Tomlinson
- Use Cluster.scheduler_info for workers= value in repr (:pr:`3058`) Matthew Rocklin
- Allow SpecCluster to scale by memory and cores (:pr:`3057`) Matthew Rocklin
- Allow full script in preload inputs (:pr:`3052`) Matthew Rocklin
- Check multiple cgroups dirs, ceil fractional cpus (:pr:`3056`) Jim Crist
- Add blurb about disabling work stealing (:pr:`3055`) Chris White
2.4.0 - 2019-09-13
- Remove six (:pr:`3045`) Matthew Rocklin
- Add missing test data to sdist tarball (:pr:`3050`) Elliott Sales de Andrade
- Use mock from unittest standard library (:pr:`3049`) Elliott Sales de Andrade
- Use cgroups resource limits to determine default threads and memory (:pr:`3039`) Jim Crist
- Move task deserialization to immediately before task execution (:pr:`3015`) James Bourbeau
- Drop joblib shim module in distributed (:pr:`3040`) John Kirkham
- Redirect configuration doc page (:pr:`3038`) Matthew Rocklin
- Support
--name 0
and--nprocs
keywords in dask-worker cli (:pr:`3037`) Matthew Rocklin - Remove lost workers from
SpecCluster.workers
(:pr:`2990`) Guillaume Eynard-Bontemps - Clean up
test_local.py::test_defaults
(:pr:`3017`) Matthew Rocklin - Replace print statement in
Queue.__init__
with debug message (:pr:`3035`) Mikhail Akimov - Set the
x_range
limit of the Meory utilization plot to memory-limit (:pr:`3034`) Matthew Rocklin - Rely on cudf codebase for cudf serialization (:pr:`2998`) Benjamin Zaitlen
- Add fallback html repr for Cluster (:pr:`3023`) Jim Crist
- Add support for zstandard compression to comms (:pr:`2970`) Abael He
- Avoid collision when using
os.environ
indashboard_link
(:pr:`3021`) Matthew Rocklin - Fix
ConnectionPool
limit handling (:pr:`3005`) byjott - Support Spec jobs that generate multiple workers (:pr:`3013`) Matthew Rocklin
- Tweak
Logs
styling (:pr:`3012`) Jim Crist - Better name for cudf deserialization function name (:pr:`3008`) Benjamin Zaitlen
- Make
spec.ProcessInterface
a valid no-op worker (:pr:`3004`) Matthew Rocklin - Return dictionaries from
new_worker_spec
rather than name/worker pairs (:pr:`3000`) Matthew Rocklin - Fix minor typo in documentation (:pr:`3002`) Mohammad Noor
- Permit more keyword options when scaling with cores and memory (:pr:`2997`) Matthew Rocklin
- Add
cuda_ipc
to UCX environment for NVLink (:pr:`2996`) Benjamin Zaitlen - Add
threads=
andmemory=
to Cluster and Client reprs (:pr:`2995`) Matthew Rocklin - Fix PyNVML initialization (:pr:`2993`) Richard J Zamora
2.3.2 - 2019-08-23
- Skip exceptions in startup information (:pr:`2991`) Jacob Tomlinson
2.3.1 - 2019-08-22
- Add support for separate external address for SpecCluster scheduler (:pr:`2963`) Jacob Tomlinson
- Defer cudf serialization/deserialization to that library (:pr:`2881`) Benjamin Zaitlen
- Workaround for hanging test now calls ucp.fin() (:pr:`2967`) Mads R. B. Kristensen
- Remove unnecessary bullet point (:pr:`2972`) Pav A
- Directly import progress from diagnostics.progressbar (:pr:`2975`) Matthew Rocklin
- Handle buffer protocol objects in ensure_bytes (:pr:`2969`) Tom Augspurger
- Fix documentatation syntax and tree (:pr:`2981`) Pav A
- Improve get_ip_interface error message when interface does not exist (:pr:`2964`) Loïc Estève
- Add cores= and memory= keywords to scale (:pr:`2974`) Matthew Rocklin
- Make workers robust to bad custom metrics (:pr:`2984`) Matthew Rocklin
2.3.0 - 2019-08-16
- Except all exceptions when checking
pynvml
(:pr:`2961`) Matthew Rocklin - Pass serialization down through small base collections (:pr:`2948`) Peter Andreas Entschev
- Use
pytest.warning(Warning)
rather thanException
(:pr:`2958`) Matthew Rocklin - Allow
server_kwargs
to override defaults in dashboard (:pr:`2955`) Bruce Merry - Update
utils_perf.py
(:pr:`2954`) Shayan Amani - Normalize names with
str
inretire_workers
(:pr:`2949`) Matthew Rocklin - Update
client.py
(:pr:`2951`) Shayan Amani - Add
GPUCurrentLoad
dashboard plots (:pr:`2944`) Matthew Rocklin - Pass GPU diagnostics from worker to scheduler (:pr:`2932`) Matthew Rocklin
- Import from
collections.abc
(:pr:`2938`) Jim Crist - Fixes Worker docstring formatting (:pr:`2939`) James Bourbeau
- Redirect setup docs to docs.dask.org (:pr:`2936`) Matthew Rocklin
- Wrap offload in
gen.coroutine
(:pr:`2934`) Matthew Rocklin - Change
TCP.close
to a coroutine to avoid task pending warning (:pr:`2930`) Matthew Rocklin - Fixup black string normalization (:pr:`2929`) Jim Crist
- Move core functionality from
SpecCluster
toCluster
(:pr:`2913`) Matthew Rocklin - Add aenter/aexit protocols to
ProcessInterface
(:pr:`2927`) Matthew Rocklin - Add real-time CPU utilization plot to dashboard (:pr:`2922`) Matthew Rocklin
- Always kill processes in clean tests, even if we don't check (:pr:`2924`) Matthew Rocklin
- Add timeouts to processes in SSH tests (:pr:`2925`) Matthew Rocklin
- Add documentation around
spec.ProcessInterface
(:pr:`2923`) Matthew Rocklin - Cleanup async warnings in tests (:pr:`2920`) Matthew Rocklin
- Give 404 when requesting nonexistent tasks or workers (:pr:`2921`) Martin Durant
- Raise informative warning when rescheduling an unknown task (:pr:`2916`) James Bourbeau
- Fix docstring (:pr:`2917`) Martin Durant
- Add keep-alive message between worker and scheduler (:pr:`2907`) Matthew Rocklin
- Rewrite
Adaptive
/SpecCluster
to support slowly arriving workers (:pr:`2904`) Matthew Rocklin - Call heartbeat rather than reconnect on disconnection (:pr:`2906`) Matthew Rocklin
2.2.0 - 2019-07-31
- Respect security configuration in LocalCluster (:pr:`2822`) Russ Bubley
- Add Nanny to worker docs (:pr:`2826`) Christian Hudon
- Don't make False add-keys report to scheduler (:pr:`2421`) tjb900
- Include type name in SpecCluster repr (:pr:`2834`) Jacob Tomlinson
- Extend prometheus metrics endpoint (:pr:`2833`) Gabriel Sailer
- Add alternative SSHCluster implementation (:pr:`2827`) Matthew Rocklin
- Dont reuse closed worker in get_worker (:pr:`2841`) Pierre Glaser
- SpecCluster: move init logic into start (:pr:`2850`) Jacob Tomlinson
- Document distributed.Reschedule in API docs (:pr:`2860`) James Bourbeau
- Add fsspec to installation of test builds (:pr:`2859`) Martin Durant
- Make await/start more consistent across Scheduler/Worker/Nanny (:pr:`2831`) Matthew Rocklin
- Add cleanup fixture for asyncio tests (:pr:`2866`) Matthew Rocklin
- Use only remote connection to scheduler in Adaptive (:pr:`2865`) Matthew Rocklin
- Add Server.finished async function (:pr:`2864`) Matthew Rocklin
- Align text and remove bullets in Client HTML repr (:pr:`2867`) Matthew Rocklin
- Test dask-scheduler --idle-timeout flag (:pr:`2862`) Matthew Rocklin
- Remove
Client.upload_environment
(:pr:`2877`) Jim Crist - Replace gen.coroutine with async/await in core (:pr:`2871`) Matthew Rocklin
- Forcefully kill all processes before each test (:pr:`2882`) Matthew Rocklin
- Cleanup Security class and configuration (:pr:`2873`) Jim Crist
- Remove unused variable in SpecCluster scale down (:pr:`2870`) Jacob Tomlinson
- Add SpecCluster ProcessInterface (:pr:`2874`) Jacob Tomlinson
- Add Log(str) and Logs(dict) classes for nice HTML reprs (:pr:`2875`) Jacob Tomlinson
- Pass Client._asynchronous to Cluster._asynchronous (:pr:`2890`) Matthew Rocklin
- Add default logs method to Spec Cluster (:pr:`2889`) Matthew Rocklin
- Add processes keyword back into clean (:pr:`2891`) Matthew Rocklin
- Update black (:pr:`2901`) Matthew Rocklin
- Move Worker.local_dir attribute to Worker.local_directory (:pr:`2900`) Matthew Rocklin
- Link from TapTools to worker info pages in dashboard (:pr:`2894`) Matthew Rocklin
- Avoid exception in Client._ensure_connected if closed (:pr:`2893`) Matthew Rocklin
- Convert Pythonic kwargs to CLI Keywords for SSHCluster (:pr:`2898`) Matthew Rocklin
- Use kwargs in CLI (:pr:`2899`) Matthew Rocklin
- Name SSHClusters by providing name= keyword to SpecCluster (:pr:`2903`) Matthew Rocklin
- Request feed of worker information from Scheduler to SpecCluster (:pr:`2902`) Matthew Rocklin
- Clear out compatibillity file (:pr:`2896`) Matthew Rocklin
- Remove future imports (:pr:`2897`) Matthew Rocklin
- Use click's show_default=True in relevant places (:pr:`2838`) Christian Hudon
- Close workers more gracefully (:pr:`2905`) Matthew Rocklin
- Close workers gracefully with --lifetime keywords (:pr:`2892`) Matthew Rocklin
- Add closing <li> tags to Client._repr_html_ (:pr:`2911`) Matthew Rocklin
- Add endline spacing in Logs._repr_html_ (:pr:`2912`) Matthew Rocklin
2.1.0 - 2019-07-08
- Fix typo that prevented error message (:pr:`2825`) Russ Bubley
- Remove
dask-mpi
(:pr:`2824`) Matthew Rocklin - Updates to use
update_graph
in task journey docs (:pr:`2821`) James Bourbeau - Fix Client repr with
memory_info=None
(:pr:`2816`) Matthew Rocklin - Fix case where key, rather than
TaskState
, could end up ints.waiting_on
(:pr:`2819`) tjb900 - Use Keyword-only arguments (:pr:`2814`) Matthew Rocklin
- Relax check for worker references in cluster context manager (:pr:`2813`) Matthew Rocklin
- Add HTTPS support for the dashboard (:pr:`2812`) Jim Crist
- Use
dask.utils.format_bytes
(:pr:`2810`) Tom Augspurger
2.0.1 - 2019-06-26
We neglected to include python_requires=
in our setup.py file, resulting in
confusion for Python 2 users who erroneously get packages for 2.0.0.
This is fixed in 2.0.1 and we have removed the 2.0.0 files from PyPI.
- Add python_requires entry to setup.py (:pr:`2807`) Matthew Rocklin
- Correctly manage tasks beyond deque limit in TaskStream plot (:pr:`2797`) Matthew Rocklin
- Fix diagnostics page for memory_limit=None (:pr:`2770`) Brett Naul
2.0.0 - 2019-06-25
- Drop support for Python 2
- Relax warnings before release (:pr:`2796`) Matthew Rocklin
- Deprecate --bokeh/--no-bokeh CLI (:pr:`2800`) Tom Augspurger
- Typo in bokeh service_kwargs for dask-worker (:pr:`2783`) Tom Augspurger
- Update command line cli options docs (:pr:`2794`) James Bourbeau
- Remove "experimental" from TLS docs (:pr:`2793`) James Bourbeau
- Add warnings around ncores= keywords (:pr:`2791`) Matthew Rocklin
- Add --version option to scheduler and worker CLI (:pr:`2782`) Tom Augspurger
- Raise when workers initialization times out (:pr:`2784`) Tom Augspurger
- Replace ncores with nthreads throughout codebase (:pr:`2758`) Matthew Rocklin
- Add unknown pytest markers (:pr:`2764`) Tom Augspurger
- Delay lookup of allowed failures. (:pr:`2761`) Tom Augspurger
- Change address -> worker in ColumnDataSource for nbytes plot (:pr:`2755`) Matthew Rocklin
- Remove module state in Prometheus Handlers (:pr:`2760`) Matthew Rocklin
- Add stress test for UCX (:pr:`2759`) Matthew Rocklin
- Add nanny logs (:pr:`2744`) Tom Augspurger
- Move some of the adaptive logic into the scheduler (:pr:`2735`) Matthew Rocklin
- Add SpecCluster.new_worker_spec method (:pr:`2751`) Matthew Rocklin
- Worker dashboard fixes (:pr:`2747`) Matthew Rocklin
- Add async context managers to scheduler/worker classes (:pr:`2745`) Matthew Rocklin
- Fix the resource key representation before sending graphs (:pr:`2733`) Michael Spiegel
- Allow user to configure whether workers are daemon. (:pr:`2739`) Caleb
- Pin pytest >=4 with pip in appveyor and python 3.5 (:pr:`2737`) Matthew Rocklin
- Add Experimental UCX Comm (:pr:`2591`) Ben Zaitlen Tom Augspurger Matthew Rocklin
- Close nannies gracefully (:pr:`2731`) Matthew Rocklin
- add kwargs to progressbars (:pr:`2638`) Manuel Garrido
- Add back LocalCluster.__repr__. (:pr:`2732`) Loïc Estève
- Move bokeh module to dashboard (:pr:`2724`) Matthew Rocklin
- Close clusters at exit (:pr:`2730`) Matthew Rocklin
- Add SchedulerPlugin TaskState example (:pr:`2622`) Matt Nicolls
- Add SpecificationCluster (:pr:`2675`) Matthew Rocklin
- Replace register_worker_callbacks with worker plugins (:pr:`2453`) Matthew Rocklin
- Proxy worker dashboards from scheduler dashboard (:pr:`2715`) Ben Zaitlen
- Add docstring to Scheduler.check_idle_saturated (:pr:`2721`) Matthew Rocklin
- Refer to LocalCluster in Client docstring (:pr:`2719`) Matthew Rocklin
- Remove special casing of Scikit-Learn BaseEstimator serialization (:pr:`2713`) Matthew Rocklin
- Fix two typos in Pub class docstring (:pr:`2714`) Magnus Nord
- Support uploading files with multiple modules (:pr:`2587`) Sam Grayson
- Change the main workers bokeh page to /status (:pr:`2689`) Ben Zaitlen
- Cleanly stop periodic callbacks in Client (:pr:`2705`) Matthew Rocklin
- Disable pan tool for the Progress, Byte Stored and Tasks Processing plot (:pr:`2703`) Mathieu Dugré
- Except errors in Nanny's memory monitor if process no longer exists (:pr:`2701`) Matthew Rocklin
- Handle heartbeat when worker has just left (:pr:`2702`) Matthew Rocklin
- Modify styling of histograms for many-worker dashboard plots (:pr:`2695`) Mathieu Dugré
- Add method to wait for n workers before continuing (:pr:`2688`) Daniel Farrell
- Support computation on delayed(None) (:pr:`2697`) Matthew Rocklin
- Cleanup localcluster (:pr:`2693`) Matthew Rocklin
- Use 'temporary-directory' from dask.config for Worker's directory (:pr:`2654`) Matthew Rocklin
- Remove support for Iterators and Queues (:pr:`2671`) Matthew Rocklin
1.28.1 - 2019-05-13
This is a small bugfix release due to a config change upstream.
- Use config accessor method for "scheduler-address" (:pr:`2676`) James Bourbeau
1.28.0 - 2019-05-08
- Add Type Attribute to TaskState (:pr:`2657`) Matthew Rocklin
- Add waiting task count to progress title bar (:pr:`2663`) James Bourbeau
- DOC: Clean up reference to cluster object (:pr:`2664`) K.-Michael Aye
- Allow scheduler to politely close workers as part of shutdown (:pr:`2651`) Matthew Rocklin
- Check direct_to_workers before using get_worker in Client (:pr:`2656`) Matthew Rocklin
- Fixed comment regarding keeping existing level if less verbose (:pr:`2655`) Brett Randall
- Add idle timeout to scheduler (:pr:`2652`) Matthew Rocklin
- Avoid deprecation warnings (:pr:`2653`) Matthew Rocklin
- Use an LRU cache for deserialized functions (:pr:`2623`) Matthew Rocklin
- Rename Worker._close to Worker.close (:pr:`2650`) Matthew Rocklin
- Add Comm closed bookkeeping (:pr:`2648`) Matthew Rocklin
- Explain LocalCluster behavior in Client docstring (:pr:`2647`) Matthew Rocklin
- Add last worker into KilledWorker exception to help debug (:pr:`2610`) @plbertrand
- Set working worker class for dask-ssh (:pr:`2646`) Martin Durant
- Add as_completed methods to docs (:pr:`2642`) Jim Crist
- Add timeout to Client._reconnect (:pr:`2639`) Jim Crist
- Limit test_spill_by_default memory, reenable it (:pr:`2633`) Peter Andreas Entschev
- Use proper address in worker -> nanny comms (:pr:`2640`) Jim Crist
- Fix deserialization of bytes chunks larger than 64MB (:pr:`2637`) Peter Andreas Entschev
1.27.1 - 2019-04-29
- Adaptive: recommend close workers when any are idle (:pr:`2330`) Michael Delgado
- Increase GC thresholds (:pr:`2624`) Matthew Rocklin
- Add interface= keyword to LocalCluster (:pr:`2629`) Matthew Rocklin
- Add worker_class argument to LocalCluster (:pr:`2625`) Matthew Rocklin
- Remove Python 2.7 from testing matrix (:pr:`2631`) Matthew Rocklin
- Add number of trials to diskutils test (:pr:`2630`) Matthew Rocklin
- Fix parameter name in LocalCluster docstring (:pr:`2626`) Loïc Estève
- Integrate stacktrace for low-level profiling (:pr:`2575`) Peter Andreas Entschev
- Apply Black to standardize code styling (:pr:`2614`) Matthew Rocklin
- added missing whitespace to start_worker cmd (:pr:`2613`) condoratberlin
- Updated logging module doc links from docs.python.org/2 to docs.python.org/3. (:pr:`2635`) Brett Randall
1.27.0 - 2019-04-12
- Add basic health endpoints to scheduler and worker bokeh. (:pr:`2607`) amerkel2
- Improved description accuracy of --memory-limit option. (:pr:`2601`) Brett Randall
- Check self.dependencies when looking at dependent tasks in memory (:pr:`2606`) deepthirajagopalan7
- Add RabbitMQ SchedulerPlugin example (:pr:`2604`) Matt Nicolls
- add resources to scheduler update_graph plugin (:pr:`2603`) Matt Nicolls
- Use ensure_bytes in serialize_error (:pr:`2588`) Matthew Rocklin
- Specify data storage explicitly from Worker constructor (:pr:`2600`) Matthew Rocklin
- Change bokeh port keywords to dashboard_address (:pr:`2589`) Matthew Rocklin
- .detach_() pytorch tensor to serialize data as numpy array. (:pr:`2586`) Muammar El Khatib
- Add warning if creating scratch directories takes a long time (:pr:`2561`) Matthew Rocklin
- Fix typo in pub-sub doc. (:pr:`2599`) Loïc Estève
- Allow return_when='FIRST_COMPLETED' in wait (:pr:`2598`) Nikos Tsaousis
- Forward kwargs through Nanny to Worker (:pr:`2596`) Brian Chu
- Use ensure_dict instead of dict (:pr:`2594`) James Bourbeau
- Specify protocol in LocalCluster (:pr:`2489`) Matthew Rocklin
1.26.1 - 2019-03-29
- Fix LocalCluster to not overallocate memory when overcommitting threads per worker (:pr:`2541`) George Sakkis
- Make closing resilient to lacking an address (:pr:`2542`) Matthew Rocklin
- fix typo in comment (:pr:`2546`) Brett Jurman
- Fix double init of prometheus metrics (:pr:`2544`) Marco Neumann
- Skip test_duplicate_clients without bokeh. (:pr:`2553`) Elliott Sales de Andrade
- Add blocked_handlers to servers (:pr:`2556`) Chris White
- Always yield Server.handle_comm coroutine (:pr:`2559`) Tom Augspurger
- Use yaml.safe_load (:pr:`2566`) Matthew Rocklin
- Fetch executables from build root. (:pr:`2551`) Elliott Sales de Andrade
- Fix Torando 6 test failures (:pr:`2570`) Matthew Rocklin
- Fix test_sync_closed_loop (:pr:`2572`) Matthew Rocklin
1.26.0 - 2019-02-25
- Update style to fix recent flake8 update (:pr:`2500`) (:pr:`2509`) Matthew Rocklin
- Fix typo in gen_cluster log message (:pr:`2503`) Loïc Estève
- Allow KeyError when closing event loop (:pr:`2498`) Matthew Rocklin
- Avoid thread testing for TCP ThreadPoolExecutor (:pr:`2510`) Matthew Rocklin
- Find Futures inside SubgraphCallable (:pr:`2505`) Jim Crist
- Avoid AttributeError when closing and sending a message (:pr:`2514`) Matthew Rocklin
- Add deprecation warning to dask_mpi.py (:pr:`2522`) Julia Kent
- Relax statistical profiling test (:pr:`2527`) Matthew Rocklin
- Support alternative --remote-dask-worker SSHCluster() and dask-ssh CLI (:pr:`2526`) Adam Beberg
- Iterate over full list of plugins in transition (:pr:`2518`) Matthew Rocklin
- Create Prometheus Endpoint (:pr:`2499`) Adam Beberg
- Use pytest.importorskip for prometheus test (:pr:`2533`) Matthew Rocklin
- MAINT skip prometheus test when no installed (:pr:`2534`) Olivier Grisel
- Fix intermittent testing failures (:pr:`2535`) Matthew Rocklin
- Avoid using nprocs keyword in dask-ssh if set to one (:pr:`2531`) Matthew Rocklin
- Bump minimum Tornado version to 5.0
1.25.3 - 2019-01-31
- Fix excess threading on missing connections (:pr:`2403`) Daniel Farrell
- Fix typo in doc (:pr:`2457`) Loïc Estève
- Start fewer but larger workers with LocalCluster (:pr:`2452`) Matthew Rocklin
- Check for non-zero
length
first inread
loop (:pr:`2465`) John Kirkham - DOC: Use of local cluster in script (:pr:`2462`) Peter Killick
- DOC/API: Signature for base class write / read (:pr:`2472`) Tom Augspurger
- Support Pytest 4 in Tests (:pr:`2478`) Adam Beberg
- Ensure async behavior in event loop with LocalCluster (:pr:`2484`) Matthew Rocklin
- Fix spurious CancelledError (:pr:`2485`) Loïc Estève
- Properly reset dask.config scheduler and shuffle when closing the client (:pr:`2475`) George Sakkis
- Make it more explict that resources are per worker. (:pr:`2470`) Loïc Estève
- Remove references to center (:pr:`2488`) Matthew Rocklin
- Expand client clearing timeout to 10s in testing (:pr:`2493`) Matthew Rocklin
- Propagate key keyword in progressbar (:pr:`2492`) Matthew Rocklin
- Use provided cluster's IOLoop if present in Client (:pr:`2494`) Matthew Rocklin
1.25.2 - 2019-01-04
- Clean up LocalCluster logging better in async mode (:pr:`2448`) Matthew Rocklin
- Add short error message if bokeh cannot be imported (:pr:`2444`) Dirk Petersen
- Add optional environment variables to Nanny (:pr:`2431`) Matthew Rocklin
- Make the direct keyword docstring entries uniform (:pr:`2441`) Matthew Rocklin
- Make LocalCluster.close async friendly (:pr:`2437`) Matthew Rocklin
- gather_dep: don't request dependencies we already found out we don't want (:pr:`2428`) tjb900
- Add parameters to Client.run docstring (:pr:`2429`) Matthew Rocklin
- Support coroutines and async-def functions in run/run_scheduler (:pr:`2427`) Matthew Rocklin
- Name threads in ThreadPoolExecutors (:pr:`2408`) Matthew Rocklin
1.25.1 - 2018-12-15
- Serialize numpy.ma.masked objects properly (:pr:`2384`) Jim Crist
- Turn off bokeh property validation in dashboard (:pr:`2387`) Jim Crist
- Fully initialize WorkerState objects (:pr:`2388`) Jim Crist
- Fix typo in scheduler docstring (:pr:`2393`) Russ Bubley
- DOC: fix typo in distributed.worker.Worker docstring (:pr:`2395`) Loïc Estève
- Remove clients and workers from event log after removal (:pr:`2394`) tjb900
- Support msgpack 0.6.0 by providing length keywords (:pr:`2399`) tjb900
- Use async-await on large messages test (:pr:`2404`) Matthew Rocklin
- Fix race condition in normalize_collection (:pr:`2386`) Jim Crist
- Fix redict collection after HighLevelGraph fix upstream (:pr:`2413`) Matthew Rocklin
- Add a blocking argument to Lock.acquire() (:pr:`2412`) Stephan Hoyer
- Fix long traceback test (:pr:`2417`) Matthew Rocklin
- Update x509 certificates to current OpenSSL standards. (:pr:`2418`) Diane Trout
1.25.0 - 2018-11-28
- Fixed the 404 error on the Scheduler Dashboard homepage (:pr:`2361`) Michael Wheeler
- Consolidate two Worker classes into one (:pr:`2363`) Matthew Rocklin
- Avoid warnings in pyarrow and msgpack (:pr:`2364`) Matthew Rocklin
- Avoid race condition in Actor's Future (:pr:`2374`) Matthew Rocklin
- Support missing packages keyword in Client.get_versions (:pr:`2379`) Matthew Rocklin
- Fixup serializing masked arrays (:pr:`2373`) Jim Crist
1.24.2 - 2018-11-15
- Add support for Bokeh 1.0 (:pr:`2348`) (:pr:`2356`) Matthew Rocklin
- Fix regression that dropped support for Tornado 4 (:pr:`2353`) Roy Wedge
- Avoid deprecation warnings (:pr:`2355`) (:pr:`2357`) Matthew Rocklin
- Fix typo in worker documentation (:pr:`2349`) Tom Rochette
1.24.1 - 2018-11-09
- Use tornado's builtin AnyThreadLoopEventPolicy (:pr:`2326`) Matthew Rocklin
- Adjust TLS tests for openssl 1.1 (:pr:`2331`) Marius van Niekerk
- Avoid setting event loop policy if within Jupyter notebook server (:pr:`2343`) Matthew Rocklin
- Add preload script to conf (:pr:`2325`) Guillaume Eynard-Bontemps
- Add serializer for Numpy masked arrays (:pr:`2335`) Peter Killick
- Use psutil.Process.oneshot (:pr:`2339`) NotSqrt
- Use worker SSL context when getting client from worker. (:pr:`2301`) Anonymous
1.24.0 - 2018-10-26
- Remove Joblib Dask Backend from codebase (:pr:`2298`) Matthew Rocklin
- Include worker tls protocol in Scheduler.restart (:pr:`2295`) Matthew Rocklin
- Adapt to new Bokeh selection for 1.0 (:pr:`2292`) Matthew Rocklin
- Add explicit retry method to Future and Client (:pr:`2299`) Matthew Rocklin
- Point to main worker page in bokeh links (:pr:`2300`) Matthew Rocklin
- Limit concurrency when gathering many times (:pr:`2303`) Matthew Rocklin
- Add tls_cluster pytest fixture (:pr:`2302`) Matthew Rocklin
- Convert ConnectionPool.open and active to properties (:pr:`2304`) Matthew Rocklin
- change export_tb to format_tb (:pr:`2306`) Eric Ma
- Redirect joblib page to dask-ml (:pr:`2307`) Matthew Rocklin
- Include unserializable object in error message (:pr:`2310`) Matthew Rocklin
- Import Mapping, Iterator, Set from collections.abc in Python 3 (:pr:`2315`) Gaurav Sheni
- Extend Client.scatter docstring (:pr:`2320`) Eric Ma
- Update for new flake8 (:pr:`2321`) Matthew Rocklin
1.23.3 - 2018-10-05
- Err in dask serialization if not a NotImplementedError (:pr:`2251`) Matthew Rocklin
- Protect against key missing from priority in GraphLayout (:pr:`2259`) Matthew Rocklin
- Do not pull data twice in Client.gather (:pr:`2263`) Adam Klein
- Add pytest fixture for cluster tests (:pr:`2262`) Matthew Rocklin
- Cleanup bokeh callbacks (:pr:`2261`) (:pr:`2278`) Matthew Rocklin
- Fix bokeh error for memory_limit=None (:pr:`2255`) Brett Naul
- Place large keywords into task graph in Client.map (:pr:`2281`) Matthew Rocklin
- Remove redundant blosc threading code from protocol.numpy (:pr:`2284`) Mike Gevaert
- Add ncores to workertable (:pr:`2289`) Matthew Rocklin
- Support upload_file on files with no extension (:pr:`2290`) Matthew Rocklin
1.23.2 - 2018-09-17
- Discard dependent rather than remove (:pr:`2250`) Matthew Rocklin
- Use dask_sphinx_theme Matthew Rocklin
- Drop the Bokeh index page (:pr:`2241`) John Kirkham
- Revert change to keep link relative (:pr:`2242`) Matthew Rocklin
- docs: Fix broken AWS link in setup.rst file (:pr:`2240`) Vladyslav Moisieienkov
- Return cancelled futures in as_completed (:pr:`2233`) Chris White
1.23.1 - 2018-09-06
- Raise informative error when mixing futures between clients (:pr:`2227`) Matthew Rocklin
- add byte_keys to unpack_remotedata call (:pr:`2232`) Matthew Rocklin
- Add documentation for gist/rawgit for get_task_stream (:pr:`2236`) Matthew Rocklin
- Quiet Client.close by waiting for scheduler stop signal (:pr:`2237`) Matthew Rocklin
- Display system graphs nicely on different screen sizes (:pr:`2239`) Derek Ludwig
- Mutate passed in workers dict in TaskStreamPlugin.rectangles (:pr:`2238`) Matthew Rocklin
1.23.0 - 2018-08-30
- Add direct_to_workers to Client Matthew Rocklin
- Add Scheduler.proxy to workers Matthew Rocklin
- Implement Actors Matthew Rocklin
- Fix tooltip (:pr:`2168`) Loïc Estève
- Fix scale / avoid returning coroutines (:pr:`2171`) Joe Hamman
- Clarify dask-worker --nprocs (:pr:`2173`) Yu Feng
- Concatenate all bytes of small messages in TCP comms (:pr:`2172`) Matthew Rocklin
- Add dashboard_link property (:pr:`2176`) Jacob Tomlinson
- Always offload to_frames (:pr:`2170`) Matthew Rocklin
- Warn if desired port is already in use (:pr:`2191`) (:pr:`2199`) Matthew Rocklin
- Add profile page for event loop thread (:pr:`2144`) Matthew Rocklin
- Use dispatch for dask serialization, also add sklearn, pytorch (:pr:`2175`) Matthew Rocklin
- Handle corner cases with busy signal (:pr:`2182`) Matthew Rocklin
- Check self.dependencies when looking at tasks in memory (:pr:`2196`) Matthew Rocklin
- Add ability to log additional custom metrics from each worker (:pr:`2169`) Loïc Estève
- Fix formatting when port is a tuple (:pr:`2204`) Loïc Estève
- Describe what ZeroMQ is (:pr:`2211`) Mike DePalatis
- Tiny typo fix (:pr:`2214`) Anderson Banihirwe
- Add Python 3.7 to travis.yml (:pr:`2203`) Matthew Rocklin
- Add plot= keyword to get_task_stream (:pr:`2198`) Matthew Rocklin
- Add support for optional versions in Client.get_versions (:pr:`2216`) Matthew Rocklin
- Add routes for solo bokeh figures in dashboard (:pr:`2185`) Matthew Rocklin
- Be resilient to missing dep after busy signal (:pr:`2217`) Matthew Rocklin
- Use CSS Grid to layout status page on the dashboard (:pr:`2213`) Derek Ludwig and Luke Canavan
- Fix deserialization of queues on main ioloop thread (:pr:`2221`) Matthew Rocklin
- Add a worker initialization function (:pr:`2201`) Guillaume Eynard-Bontemps
- Collapse navbar in dashboard (:pr:`2223`) Luke Canavan
1.22.1 - 2018-08-03
- Add worker_class= keyword to Nanny to support different worker types (:pr:`2147`) Martin Durant
- Cleanup intermittent worker failures (:pr:`2152`) (:pr:`2146`) Matthew Rocklin
- Fix msgpack PendingDeprecationWarning for encoding='utf-8' (:pr:`2153`) Olivier Grisel
- Make bokeh coloring deterministic using hash function (:pr:`2143`) Matthew Rocklin
- Allow client to query the task stream plot (:pr:`2122`) Matthew Rocklin
- Use PID and counter in thread names (:pr:`2084`) (:pr:`2128`) Dror Birkman
- Test that worker restrictions are cleared after cancellation (:pr:`2107`) Matthew Rocklin
- Expand resources in graph_to_futures (:pr:`2131`) Matthew Rocklin
- Add custom serialization support for pyarrow (:pr:`2115`) Dave Hirschfeld
- Update dask-scheduler cli help text for preload (:pr:`2120`) Matt Nicolls
- Added another nested parallelism test (:pr:`1710`) Tom Augspurger
- insert newline by default after TextProgressBar (:pr:`1976`) Phil Tooley
- Retire workers from scale (:pr:`2104`) Matthew Rocklin
- Allow worker to refuse data requests with busy signal (:pr:`2092`) Matthew Rocklin
- Don't forget released keys (:pr:`2098`) Matthew Rocklin
- Update example for stopping a worker (:pr:`2088`) John Kirkham
- removed hardcoded value of memory terminate fraction from a log message (:pr:`2096`) Bartosz Marcinkowski
- Adjust worker doc after change in config file location and treatment (:pr:`2094`) Aurélien Ponte
- Prefer gathering data from same host (:pr:`2090`) Matthew Rocklin
- Handle exceptions on deserialized comm with text error (:pr:`2093`) Matthew Rocklin
- Fix typo in docstring (:pr:`2087`) Loïc Estève
- Provide communication context to serialization functions (:pr:`2054`) Matthew Rocklin
- Allow name to be explicitly passed in publish_dataset (:pr:`1995`) Marius van Niekerk
- Avoid accessing Worker.scheduler_delay around yield point (:pr:`2074`) Matthew Rocklin
- Support TB and PB in format bytes (:pr:`2072`) Matthew Rocklin
- Add test for as_completed for loops in Python 2 (:pr:`2071`) Matthew Rocklin
- Allow adaptive to exist without a cluster (:pr:`2064`) Matthew Rocklin
- Have worker data transfer wait until recipient acknowledges (:pr:`2052`) Matthew Rocklin
- Support async def functions in Client.sync (:pr:`2070`) Matthew Rocklin
- Add asynchronous parameter to docstring of LocalCluster Matthew Rocklin
- Normalize address before comparison (:pr:`2066`) Tom Augspurger
- Use ConnectionPool for Worker.scheduler Matthew Rocklin
- Avoid reference cycle in str_graph Matthew Rocklin
- Pull data outside of while loop in gather (:pr:`2059`) Matthew Rocklin
1.22.0 - 2018-06-14
- Overhaul configuration (:pr:`1948`) Matthew Rocklin
- Replace get= keyword with scheduler= (:pr:`1959`) Matthew Rocklin
- Use tuples in msgpack (:pr:`2000`) Matthew Rocklin and Marius van Niekerk
- Unify handling of high-volume connections (:pr:`1970`) Matthew Rocklin
- Automatically scatter large arguments in joblib connector (:pr:`2020`) (:pr:`2030`) Olivier Grisel
- Turn click Python 3 locales failure into a warning (:pr:`2001`) Matthew Rocklin
- Rely on dask implementation of sizeof (:pr:`2042`) Matthew Rocklin
- Replace deprecated workers.iloc with workers.values() (:pr:`2013`) Grant Jenks
- Introduce serialization families (:pr:`1912`) Matthew Rocklin
- Add PubSub (:pr:`1999`) Matthew Rocklin
- Add Dask stylesheet to documentation Matthew Rocklin
- Avoid recomputation on partially-complete results (:pr:`1840`) Matthew Rocklin
- Use sys.prefix in popen for testing (:pr:`1954`) Matthew Rocklin
- Include yaml files in manifest Matthew Rocklin
- Use self.sync so Client.processing works in asynchronous context (:pr:`1962`) Henry Doupe
- Fix bug with bad repr on closed client (:pr:`1965`) Matthew Rocklin
- Parse --death-timeout keyword in dask-worker (:pr:`1967`) Matthew Rocklin
- Support serializers in BatchedSend (:pr:`1964`) Matthew Rocklin
- Use normal serialization mechanisms to serialize published datasets (:pr:`1972`) Matthew Rocklin
- Add security support to LocalCluster. (:pr:`1855`) Marius van Niekerk
- add ConnectionPool.remove method (:pr:`1977`) Tony Lorenzo
- Cleanly close workers when scheduler closes (:pr:`1981`) Matthew Rocklin
- Add .pyz support in upload_file (:pr:`1781`) @bmaisson
- add comm to packages (:pr:`1980`) Matthew Rocklin
- Replace dask.set_options with dask.config.set Matthew Rocklin
- Exclude versions of sortedcontainers which do not have .iloc. (:pr:`1993`) Russ Bubley
- Exclude gc statistics under PyPy (:pr:`1997`) Marius van Niekerk
- Manage recent config and dataframe changes in dask (:pr:`2009`) Matthew Rocklin
- Cleanup lingering clients in tests (:pr:`2012`) Matthew Rocklin
- Use timeouts during Client._ensure_connected (:pr:`2011`) Martin Durant
- Avoid reference cycle in joblib backend (:pr:`2014`) Matthew Rocklin, also Olivier Grisel
- DOC: fixed test example (:pr:`2017`) Tom Augspurger
- Add worker_key parameter to Adaptive (:pr:`1992`) Matthew Rocklin
- Prioritize tasks with their true keys, before stringifying (:pr:`2006`) Matthew Rocklin
- Serialize worker exceptions through normal channels (:pr:`2016`) Matthew Rocklin
- Include exception in progress bar (:pr:`2028`) Matthew Rocklin
- Avoid logging orphaned futures in All (:pr:`2008`) Matthew Rocklin
- Don't use spill-to-disk dictionary if we're not spilling to disk Matthew Rocklin
- Only avoid recomputation if key exists (:pr:`2036`) Matthew Rocklin
- Use client connection and serialization arguments in progress (:pr:`2035`) Matthew Rocklin
- Rejoin worker client on closing context manager (:pr:`2041`) Matthew Rocklin
- Avoid forgetting erred tasks when losing dependencies (:pr:`2047`) Matthew Rocklin
- Avoid collisions in graph_layout (:pr:`2050`) Matthew Rocklin
- Avoid recursively calling bokeh callback in profile plot (:pr:`2048`) Matthew Rocklin
1.21.8 - 2018-05-03
- Remove errant print statement (:pr:`1957`) Matthew Rocklin
- Only add reevaluate_occupancy callback once (:pr:`1953`) Tony Lorenzo
1.21.7 - 2018-05-02
- Newline needed for doctest rendering (:pr:`1917`) Loïc Estève
- Support Client._repr_html_ when in async mode (:pr:`1909`) Matthew Rocklin
- Add parameters to dask-ssh command (:pr:`1910`) Irene Rodriguez
- Santize get_dataset trace (:pr:`1888`) John Kirkham
- Fix bug where queues would not clean up cleanly (:pr:`1922`) Matthew Rocklin
- Delete cached file safely in upload file (:pr:`1921`) Matthew Rocklin
- Accept KeyError when closing tornado IOLoop in tests (:pr:`1937`) Matthew Rocklin
- Quiet the client and scheduler when gather(..., errors='skip') (:pr:`1936`) Matthew Rocklin
- Clarify couldn't gather keys warning (:pr:`1942`) Kenneth Koski
- Support submit keywords in joblib (:pr:`1947`) Matthew Rocklin
- Avoid use of external resources in bokeh server (:pr:`1934`) Matthew Rocklin
- Drop __contains__ from Datasets (:pr:`1889`) John Kirkham
- Fix bug with queue timeouts (:pr:`1950`) Matthew Rocklin
- Replace msgpack-python by msgpack (:pr:`1927`) Loïc Estève
1.21.6 - 2018-04-06
- Fix numeric environment variable configuration (:pr:`1885`) Joseph Atkins-Kurkish
- support bytearrays in older lz4 library (:pr:`1886`) Matthew Rocklin
- Remove started timeout in nanny (:pr:`1852`) Matthew Rocklin
- Don't log errors in sync (:pr:`1894`) Matthew Rocklin
- downgrade stale lock warning to info logging level (:pr:`1890`) Matthew Rocklin
- Fix
UnboundLocalError
forkey
(:pr:`1900`) John Kirkham - Resolve deployment issues in Python 2 (:pr:`1905`) Matthew Rocklin
- Support retries and priority in Client.get method (:pr:`1902`) Matthew Rocklin
- Add additional attributes to task page if applicable (:pr:`1901`) Matthew Rocklin
- Add count method to as_completed (:pr:`1897`) Matthew Rocklin
- Extend default timeout to 10s (:pr:`1904`) Matthew Rocklin
1.21.5 - 2018-03-31
- Increase default allowable tick time to 3s (:pr:`1854`) Matthew Rocklin
- Handle errant workers when another worker has data (:pr:`1853`) Matthew Rocklin
- Close multiprocessing queue in Nanny to reduce open file descriptors (:pr:`1862`) Matthew Rocklin
- Extend nanny started timeout to 30s, make configurable (:pr:`1865`) Matthew Rocklin
- Comment out the default config file (:pr:`1871`) Matthew Rocklin
- Update to fix bokeh 0.12.15 update errors (:pr:`1872`) Matthew Rocklin
- Downgrade Event Loop unresponsive warning to INFO level (:pr:`1870`) Matthew Rocklin
- Add fifo timeout to control priority generation (:pr:`1828`) Matthew Rocklin
- Add retire_workers API to Client (:pr:`1876`) Matthew Rocklin
- Catch NoSuchProcess error in Nanny.memory_monitor (:pr:`1877`) Matthew Rocklin
- Add uid to nanny queue communitcations (:pr:`1880`) Matthew Rocklin
1.21.4 - 2018-03-21
- Avoid passing bytearrays to snappy decompression (:pr:`1831`) Matthew Rocklin
- Specify IOLoop in Adaptive (:pr:`1841`) Matthew Rocklin
- Use connect-timeout config value throughout client (:pr:`1839`) Matthew Rocklin
- Support direct= keyword argument in Client.get (:pr:`1845`) Matthew Rocklin
1.21.3 - 2018-03-08
- Add cluster superclass and improve adaptivity (:pr:`1813`) Matthew Rocklin
- Fixup tests and support Python 2 for Tornado 5.0 (:pr:`1818`) Matthew Rocklin
- Fix bug in recreate_error when dependencies are dropped (:pr:`1815`) Matthew Rocklin
- Add worker time to live in Scheduler (:pr:`1811`) Matthew Rocklin
- Scale adaptive based on total_occupancy (:pr:`1807`) Matthew Rocklin
- Support calling compute within worker_client (:pr:`1814`) Matthew Rocklin
- Add percentage to profile plot (:pr:`1817`) Brett Naul
- Overwrite option for remote python in dask-ssh (:pr:`1812`) Sven Kreiss
1.21.2 - 2018-03-05
- Fix bug where we didn't check idle/saturated when stealing (:pr:`1801`) Matthew Rocklin
- Fix bug where client was noisy when scheduler closed unexpectedly (:pr:`1806`) Matthew Rocklin
- Use string-based timedeltas (like
'500 ms'
) everywhere (:pr:`1804`) Matthew Rocklin - Keep logs in scheduler and worker even if silenced (:pr:`1803`) Matthew Rocklin
- Support minimum, maximum, wait_count keywords in Adaptive (:pr:`1797`) Jacob Tomlinson and Matthew Rocklin
- Support async protocols for LocalCluster, replace start= with asynchronous= (:pr:`1798`) Matthew Rocklin
- Avoid restarting workers when nanny waits on scheduler (:pr:`1793`) Matthew Rocklin
- Use
IOStream.read_into()
when available (:pr:`1477`) Antoine Pitrou - Reduce LocalCluster logging threshold from CRITICAL to WARN (:pr:`1785`) Andy Jones
- Add futures_of to API docs (:pr:`1783`) John Kirkham
- Make diagnostics link in client configurable (:pr:`1810`) Matthew Rocklin
1.21.1 - 2018-02-22
- Fixed an uncaught exception in
distributed.joblib
with aLocalCluster
using only threads (:issue:`1775`) Tom Augspurger - Format bytes in info worker page (:pr:`1752`) Matthew Rocklin
- Add pass-through arguments for scheduler/worker --preload modules. (:pr:`1634`) Alexander Ford
- Use new LZ4 API (:pr:`1757`) Thrasibule
- Replace dask.optimize with dask.optimization (:pr:`1754`) Matthew Rocklin
- Add graph layout engine and bokeh plot (:pr:`1756`) Matthew Rocklin
- Only expand name with --nprocs if name exists (:pr:`1776`) Matthew Rocklin
- specify IOLoop for stealing PeriodicCallback (:pr:`1777`) Matthew Rocklin
- Fixed distributed.joblib with no processes Tom Augspurger
- Use set.discard to avoid KeyErrors in stealing (:pr:`1766`) Matthew Rocklin
- Avoid KeyError when task has been released during steal (:pr:`1765`) Matthew Rocklin
- Add versions routes to avoid the use of run in Client.get_versions (:pr:`1773`) Matthew Rocklin
- Add write_scheduler_file to Client (:pr:`1778`) Joe Hamman
- Default host to tls:// if tls information provided (:pr:`1780`) Matthew Rocklin
1.21.0 - 2018-02-09
- Refactor scheduler to use TaskState objects rather than dictionaries (:pr:`1594`) Antoine Pitrou
- Plot CPU fraction of total in workers page (:pr:`1624`) Matthew Rocklin
- Use thread CPU time in Throttled GC (:pr:`1625`) Antoine Pitrou
- Fix bug with
memory_limit=None
(:pr:`1639`) Matthew Rocklin - Add futures_of to top level api (:pr:`1646`) Matthew Rocklin
- Warn on serializing large data in Client (:pr:`1636`) Matthew Rocklin
- Fix intermittent windows failure when removing lock file (:pr:`1652`) Antoine Pitrou
- Add diagnosis and logging of poor GC Behavior (:pr:`1635`) Antoine Pitrou
- Add client-scheduler heartbeats (:pr:`1657`) Matthew Rocklin
- Return dictionary of worker info in
retire_workers
(:pr:`1659`) Matthew Rocklin - Ensure dumps_function works with unhashable functions (:pr:`1662`) Matthew Rocklin
- Collect client name ids rom client-name config variable (:pr:`1664`) Matthew Rocklin
- Allow simultaneous use of --name and --nprocs in dask-worker (:pr:`1665`) Matthew Rocklin
- Add support for grouped adaptive scaling and adaptive behavior overrides (:pr:`1632`) Alexander Ford
- Share scheduler RPC between worker and client (:pr:`1673`) Matthew Rocklin
- Allow
retries=
in ClientExecutor (:pr:`1672`) @rqx - Improve documentation for get_client and dask.compute examples (:pr:`1638`) Scott Sievert
- Support DASK_SCHEDULER_ADDRESS environment variable in worker (:pr:`1680`) Matthew Rocklin
- Support tuple-keys in retries (:pr:`1681`) Matthew Rocklin
- Use relative links in bokeh dashboard (:pr:`1682`) Matthew Rocklin
- Make message log length configurable, default to zero (:pr:`1691`) Matthew Rocklin
- Deprecate
Client.shutdown
(:pr:`1699`) Matthew Rocklin - Add warning in configuration docs to install pyyaml (:pr:`1701`) Cornelius Riemenschneider
- Handle nested parallelism in distributed.joblib (:pr:`1705`) Tom Augspurger
- Don't wait for Worker.executor to shutdown cleanly when restarting process (:pr:`1708`) Matthew Rocklin
- Add support for user defined priorities (:pr:`1651`) Matthew Rocklin
- Catch and log OSErrors around worker lock files (:pr:`1714`) Matthew Rocklin
- Remove worker prioritization. Coincides with changes to dask.order (:pr:`1730`) Matthew Rocklin
- Use process-measured memory rather than nbytes in Bokeh dashboard (:pr:`1737`) Matthew Rocklin
- Enable serialization of Locks (:pr:`1738`) Matthew Rocklin
- Support Tornado 5 beta (:pr:`1735`) Matthew Rocklin
- Cleanup remote_magic client cache after tests (:pr:`1743`) Min RK
- Allow service ports to be specified as (host, port) (:pr:`1744`) Bruce Merry
1.20.2 - 2017-12-07
- Clear deque handlers after each test (:pr:`1586`) Antoine Pitrou
- Handle deserialization in FutureState.set_error (:pr:`1592`) Matthew Rocklin
- Add process leak checker to tests (:pr:`1596`) Antoine Pitrou
- Customize process title for subprocess (:pr:`1590`) Antoine Pitrou
- Make linting a separate CI job (:pr:`1599`) Antoine Pitrou
- Fix error from get_client() with no global client (:pr:`1595`) Daniel Li
- Remove Worker.host_health, correct WorkerTable metrics (:pr:`1600`) Matthew Rocklin
- Don't mark tasks as suspicious when retire_workers called. Addresses (:pr:`1607`) Russ Bubley
- Do not include processing workers in workers_to_close (:pr:`1609`) Russ Bubley
- Disallow simultaneous scale up and down in Adaptive (:pr:`1608`) Russ Bubley
- Parse bytestrings in --memory-limit (:pr:`1615`) Matthew Rocklin
- Use environment variable for scheduler address if present (:pr:`1610`) Matthew Rocklin
- Fix deprecation warning from logger.warn (:pr:`1616`) Brett Naul
1.20.1 - 2017-11-26
- Wrap
import ssl
statements with try-except block for ssl-crippled environments, (:pr:`1570`) Xander Johnson - Support zero memory-limit in Nanny (:pr:`1571`) Matthew Rocklin
- Avoid PeriodicCallback double starts (:pr:`1573`) Matthew Rocklin
- Add disposable workspace facility (:pr:`1543`) Antoine Pitrou
- Use format_time in task_stream plots (:pr:`1575`) Matthew Rocklin
- Avoid delayed finalize calls in compute (:pr:`1577`) Matthew Rocklin
- Doc fix about secede (:pr:`1583`) Scott Sievert
- Add tracemalloc option when tracking test leaks (:pr:`1585`) Antoine Pitrou
- Add JSON routes to Bokeh server (:pr:`1584`) Matthew Rocklin
- Handle exceptions cleanly in Variables and Queues (:pr:`1580`) Matthew Rocklin
1.20.0 - 2017-11-17
- Drop use of pandas.msgpack (:pr:`1473`) Matthew Rocklin
- Add methods to get/set scheduler metadata Matthew Rocklin
- Add distributed lock Matthew Rocklin
- Add reschedule exception for worker tasks Matthew Rocklin
- Fix
nbytes()
forbytearrays
Matthew Rocklin - Capture scheduler and worker logs Matthew Rocklin
- Garbage collect after data eviction on high worker memory usage (:pr:`1488`) Olivier Grisel
- Add scheduler HTML routes to bokeh server (:pr:`1478`) (:pr:`1514`) Matthew Rocklin
- Add pytest plugin to test for resource leaks (:pr:`1499`) Antoine Pitrou
- Improve documentation for scheduler states (:pr:`1498`) Antoine Pitrou
- Correct warn_if_longer timeout in ThrottledGC (:pr:`1496`) Fabian Keller
- Catch race condition in as_completed on cancelled futures (:pr:`1507`) Matthew Rocklin
- Transactional work stealing (:pr:`1489`) (:pr:`1528`) Matthew Rocklin
- Avoid forkserver in PyPy (:pr:`1509`) Matthew Rocklin
- Add dict access to get/set datasets (:pr:`1508`) Mike DePalatis
- Support Tornado 5 (:pr:`1509`) (:pr:`1512`) (:pr:`1518`) (:pr:`1534`) Antoine Pitrou
- Move thread_state in Dask (:pr:`1523`) Jim Crist
- Use new Dask collections interface (:pr:`1513`) Matthew Rocklin
- Add nanny flag to dask-mpi Matthew Rocklin
- Remove JSON-based HTTP servers Matthew Rocklin
- Avoid doing I/O in repr/str (:pr:`1536`) Matthew Rocklin
- Fix URL for MPI4Py project (:pr:`1546`) Ian Hopkinson
- Allow automatic retries of a failed task (:pr:`1524`) Antoine Pitrou
- Clean and accelerate tests (:pr:`1548`) (:pr:`1549`) (:pr:`1552`) (:pr:`1553`) (:pr:`1560`) (:pr:`1564`) Antoine Pitrou
- Move HDFS functionality to the hdfs3 library (:pr:`1561`) Jim Crist
- Fix bug when using events page with no events (:pr:`1562`) @rbubley
- Improve diagnostic naming of tasks within tuples (:pr:`1566`) Kelvyn Yang
1.19.3 - 2017-10-16
- Handle None case in profile.identity (:pr:`1456`)
- Asyncio rewrite (:pr:`1458`)
- Add rejoin function partner to secede (:pr:`1462`)
- Nested compute (:pr:`1465`)
- Use LooseVersion when comparing Bokeh versions (:pr:`1470`)
1.19.2 - 2017-10-06
- as_completed doesn't block on cancelled futures (:pr:`1436`)
- Notify waiting threads/coroutines on cancellation (:pr:`1438`)
- Set Future(inform=True) as default (:pr:`1437`)
- Rename Scheduler.transition_story to story (:pr:`1445`)
- Future uses default client by default (:pr:`1449`)
- Add keys= keyword to Client.call_stack (:pr:`1446`)
- Add get_current_task to worker (:pr:`1444`)
- Ensure that Client remains asynchornous before ioloop starts (:pr:`1452`)
- Remove "click for worker page" in bokeh plot (:pr:`1453`)
- Add Client.current() (:pr:`1450`)
- Clean handling of restart timeouts (:pr:`1442`)
1.19.1 - September 25th, 2017
- Fix tool issues with TaskStream plot (:pr:`1425`)
- Move profile module to top level (:pr:`1423`)
1.19.0 - September 24th, 2017
- Avoid storing messages in message log (:pr:`1361`)
- fileConfig does not disable existing loggers (:pr:`1380`)
- Offload upload_file disk I/O to separate thread (:pr:`1383`)
- Add missing SSLContext (:pr:`1385`)
- Collect worker thread information from sys._curent_frames (:pr:`1387`)
- Add nanny timeout (:pr:`1395`)
- Restart worker if memory use goes above 95% (:pr:`1397`)
- Track workers memory use with psutil (:pr:`1398`)
- Track scheduler delay times in workers (:pr:`1400`)
- Add time slider to profile plot (:pr:`1403`)
- Change memory-limit keyword to refer to maximum number of bytes (:pr:`1405`)
- Add
cancel(force=)
keyword (:pr:`1408`)
1.18.2 - September 2nd, 2017
- Silently pass on cancelled futures in as_completed (:pr:`1366`)
- Fix unicode keys error in Python 2 (:pr:`1370`)
- Support numeric worker names
- Add dask-mpi executable (:pr:`1367`)
1.18.1 - August 25th, 2017
- Clean up forgotten keys in fire-and-forget workloads (:pr:`1250`)
- Handle missing extensions (:pr:`1263`)
- Allow recreate_exception on persisted collections (:pr:`1253`)
- Add asynchronous= keyword to blocking client methods (:pr:`1272`)
- Restrict to horizontal panning in bokeh plots (:pr:`1274`)
- Rename client.shutdown to client.close (:pr:`1275`)
- Avoid blocking on event loop (:pr:`1270`)
- Avoid cloudpickle errors for Client.get_versions (:pr:`1279`)
- Yield on Tornado IOStream.write futures (:pr:`1289`)
- Assume async behavior if inside a sync statement (:pr:`1284`)
- Avoid error messages on closing (:pr:`1297`), (:pr:`1296`) (:pr:`1318`) (:pr:`1319`)
- Add timeout= keyword to get_client (:pr:`1290`)
- Respect timeouts when restarting (:pr:`1304`)
- Clean file descriptor and memory leaks in tests (:pr:`1317`)
- Deprecate Executor (:pr:`1302`)
- Add timeout to ThreadPoolExecutor.shutdown (:pr:`1330`)
- Clean up AsyncProcess handling (:pr:`1324`)
- Allow unicode keys in Python 2 scheduler (:pr:`1328`)
- Avoid leaking stolen data (:pr:`1326`)
- Improve error handling on failed nanny starts (:pr:`1337`), (:pr:`1331`)
- Make Adaptive more flexible
- Support
--contact-address
and--listen-address
in worker (:pr:`1278`) - Remove old dworker, dscheduler executables (:pr:`1355`)
- Exit workers if nanny process fails (:pr:`1345`)
- Auto pep8 and flake (:pr:`1353`)
1.18.0 - July 8th, 2017
- Multi-threading safety (:pr:`1191`), (:pr:`1228`), (:pr:`1229`)
- Improve handling of byte counting (:pr:`1198`) (:pr:`1224`)
- Add get_client, secede functions, refactor worker-client relationship (:pr:`1201`)
- Allow logging configuraiton using logging.dictConfig() (:pr:`1206`) (:pr:`1211`)
- Offload serialization and deserialization to separate thread (:pr:`1218`)
- Support fire-and-forget tasks (:pr:`1221`)
- Support bytestrings as keys (for Julia) (:pr:`1234`)
- Resolve testing corner-cases (:pr:`1236`), (:pr:`1237`), (:pr:`1240`), (:pr:`1241`), (:pr:`1242`), (:pr:`1244`)
- Automatic use of scatter/gather(direct=True) in more cases (:pr:`1239`)
1.17.1 - June 14th, 2017
- Remove Python 3.4 testing from travis-ci (:pr:`1157`)
- Remove ZMQ Support (:pr:`1160`)
- Fix memoryview nbytes issue in Python 2.7 (:pr:`1165`)
- Re-enable counters (:pr:`1168`)
- Improve scheduler.restart (:pr:`1175`)
1.17.0 - June 9th, 2017
- Reevaluate worker occupancy periodically during scheduler downtime (:pr:`1038`) (:pr:`1101`)
- Add
AioClient
asyncio-compatible client API (:pr:`1029`) (:pr:`1092`) (:pr:`1099`) - Update Keras serializer (:pr:`1067`)
- Support TLS/SSL connections for security (:pr:`866`) (:pr:`1034`)
- Always create new worker directory when passed
--local-directory
(:pr:`1079`) - Support pre-scattering data when using joblib frontent (:pr:`1022`)
- Make workers more robust to failure of
sizeof
function (:pr:`1108`) and writing to disk (:pr:`1096`) - Add
is_empty
andupdate
methods toas_completed
(:pr:`1113`) - Remove
_get
coroutine and replace withget(..., sync=False)
(:pr:`1109`) - Improve API compatibility with async/await syntax (:pr:`1115`) (:pr:`1124`)
- Add distributed Queues (:pr:`1117`) and shared Variables (:pr:`1128`) to enable inter-client coordination
- Support direct client-to-worker scattering and gathering (:pr:`1130`) as well as performance enhancements when scattering data
- Style improvements for bokeh web dashboards (:pr:`1126`) (:pr:`1141`) as well as a removal of the external bokeh process
- HTML reprs for Future and Client objects (:pr:`1136`)
- Support nested collections in client.compute (:pr:`1144`)
- Use normal client API in asynchronous mode (:pr:`1152`)
- Remove old distributed.collections submodule (:pr:`1153`)
1.16.3 - May 5th, 2017
- Add bokeh template files to MANIFEST (:pr:`1063`)
- Don't set worker_client.get as default get (:pr:`1061`)
- Clean up logging on Client().shutdown() (:pr:`1055`)
1.16.2 - May 3rd, 2017
- Support
async with Client
syntax (:pr:`1053`) - Use internal bokeh server for default diagnostics server (:pr:`1047`)
- Improve styling of bokeh plots when empty (:pr:`1046`) (:pr:`1037`)
- Support efficient serialization for sparse arrays (:pr:`1040`)
- Prioritize newly arrived work in worker (:pr:`1035`)
- Prescatter data with joblib backend (:pr:`1022`)
- Make client.restart more robust to worker failure (:pr:`1018`)
- Support preloading a module or script in dask-worker or dask-scheduler processes (:pr:`1016`)
- Specify network interface in command line interface (:pr:`1007`)
- Client.scatter supports a single element (:pr:`1003`)
- Use blosc compression on all memoryviews passing through comms (:pr:`998`)
- Add concurrent.futures-compatible Executor (:pr:`997`)
- Add as_completed.batches method and return results (:pr:`994`) (:pr:`971`)
- Allow worker_clients to optionally stay within the thread pool (:pr:`993`)
- Add bytes-stored and tasks-processing diagnostic histograms (:pr:`990`)
- Run supports non-msgpack-serializable results (:pr:`965`)
1.16.1 - March 22nd, 2017
- Use inproc transport in LocalCluster (:pr:`919`)
- Add structured and queryable cluster event logs (:pr:`922`)
- Use connection pool for inter-worker communication (:pr:`935`)
- Robustly shut down spawned worker processes at shutdown (:pr:`928`)
- Worker death timeout (:pr:`940`)
- More visual reporting of exceptions in progressbar (:pr:`941`)
- Render disk and serialization events to task stream visual (:pr:`943`)
- Support async for / await protocol (:pr:`952`)
- Ensure random generators are re-seeded in worker processes (:pr:`953`)
- Upload sourcecode as zip module (:pr:`886`)
- Replay remote exceptions in local process (:pr:`894`)
1.16.0 - February 24th, 2017
- First come first served priorities on client submissions (:pr:`840`)
- Can specify Bokeh internal ports (:pr:`850`)
- Allow stolen tasks to return from either worker (:pr:`853`), (:pr:`875`)
- Add worker resource constraints during execution (:pr:`857`)
- Send small data through Channels (:pr:`858`)
- Better estimates for SciPy sparse matrix memory costs (:pr:`863`)
- Avoid stealing long running tasks (:pr:`873`)
- Maintain fortran ordering of NumPy arrays (:pr:`876`)
- Add
--scheduler-file
keyword to dask-scheduler (:pr:`877`) - Add serializer for Keras models (:pr:`878`)
- Support uploading modules from zip files (:pr:`886`)
- Improve titles of Bokeh dashboards (:pr:`895`)
1.15.2 - January 27th, 2017
- Fix a bug where arrays with large dtypes or shapes were being improperly compressed (:pr:`830` :pr:`832` :pr:`833`)
- Extend
as_completed
to accept new futures during iteration (:pr:`829`) - Add
--nohost
keyword todask-ssh
startup utility (:pr:`827`) - Support scheduler shutdown of remote workers, useful for adaptive clusters (:pr: 811 :pr:`816` :pr:`821`)
- Add
Client.run_on_scheduler
method for running debug functions on the scheduler (:pr:`808`)
1.15.1 - January 11th, 2017
- Make compatibile with Bokeh 0.12.4 (:pr:`803`)
- Avoid compressing arrays if not helpful (:pr:`777`)
- Optimize inter-worker data transfer (:pr:`770`) (:pr:`790`)
- Add --local-directory keyword to worker (:pr:`788`)
- Enable workers to arrive to the cluster with their own data. Useful if a worker leaves and comes back (:pr:`785`)
- Resolve thread safety bug when using local_client (:pr:`802`)
- Resolve scheduling issues in worker (:pr:`804`)
1.15.0 - January 2nd, 2017
- Major Worker refactor (:pr:`704`)
- Major Scheduler refactor (:pr:`717`) (:pr:`722`) (:pr:`724`) (:pr:`742`) (:pr:`743`
- Add
check
(default isFalse
) option toClient.get_versions
to raise if the versions don't match on client, scheduler & workers (:pr:`664`) Future.add_done_callback
executes in separate thread (:pr:`656`)- Clean up numpy serialization (:pr:`670`)
- Support serialization of Tornado v4.5 coroutines (:pr:`673`)
- Use CPickle instead of Pickle in Python 2 (:pr:`684`)
- Use Forkserver rather than Fork on Unix in Python 3 (:pr:`687`)
- Support abstract resources for per-task constraints (:pr:`694`) (:pr:`720`) (:pr:`737`)
- Add TCP timeouts (:pr:`697`)
- Add embedded Bokeh server to workers (:pr:`709`) (:pr:`713`) (:pr:`738`)
- Add embedded Bokeh server to scheduler (:pr:`724`) (:pr:`736`) (:pr:`738`)
- Add more precise timers for Windows (:pr:`713`)
- Add Versioneer (:pr:`715`)
- Support inter-client channels (:pr:`729`) (:pr:`749`)
- Scheduler Performance improvements (:pr:`740`) (:pr:`760`)
- Improve load balancing and work stealing (:pr:`747`) (:pr:`754`) (:pr:`757`)
- Run Tornado coroutines on workers
- Avoid slow sizeof call on Pandas dataframes (:pr:`758`)
1.14.3 - November 13th, 2016
- Remove custom Bokeh export tool that implicitly relied on nodejs (:pr:`655`)
- Clean up scheduler logging (:pr:`657`)
1.14.2 - November 11th, 2016
- Support more numpy dtypes in custom serialization, (:pr:`627`), (:pr:`630`), (:pr:`636`)
- Update Bokeh plots (:pr:`628`)
- Improve spill to disk heuristics (:pr:`633`)
- Add Export tool to Task Stream plot
- Reverse frame order in loads for very many frames (:pr:`651`)
- Add timeout when waiting on write (:pr:`653`)
1.14.0 - November 3rd, 2016
- Add
Client.get_versions()
function to return software and package information from the scheduler, workers, and client (:pr:`595`) - Improved windows support (:pr:`577`) (:pr:`590`) (:pr:`583`) (:pr:`597`)
- Clean up rpc objects explicitly (:pr:`584`)
- Normalize collections against known futures (:pr:`587`)
- Add key= keyword to map to specify keynames (:pr:`589`)
- Custom data serialization (:pr:`606`)
- Refactor the web interface (:pr:`608`) (:pr:`615`) (:pr:`621`)
- Allow user-supplied Executor in Worker (:pr:`609`)
- Pass Worker kwargs through LocalCluster
1.13.3 - October 15th, 2016
- Schedulers can retire workers cleanly
- Add
Future.add_done_callback
forconcurrent.futures
compatibility - Update web interface to be consistent with Bokeh 0.12.3
- Close streams explicitly, avoiding race conditions and supporting more robust restarts on Windows.
- Improved shuffled performance for dask.dataframe
- Add adaptive allocation cluster manager
- Reduce administrative overhead when dealing with many workers
dask-ssh --log-directory .
no longer errors- Microperformance tuning for the scheduler
1.13.2
- Revert dask_worker to use fork rather than subprocess by default
- Scatter retains type information
- Bokeh always uses subprocess rather than spawn
1.13.1
- Fix critical Windows error with dask_worker executable
1.13.0
- Rename Executor to Client (:pr:`492`)
- Add
--memory-limit
option todask-worker
, enabling spill-to-disk behavior when running out of memory (:pr:`485`) - Add
--pid-file
option to dask-worker and--dask-scheduler
(:pr:`496`) - Add
upload_environment
function to distribute conda environments. This is experimental, undocumented, and may change without notice. (:pr:`494`) - Add
workers=
keyword argument toClient.compute
andClient.persist
, supporting location-restricted workloads with Dask collections (:pr:`484`) - Add
upload_environment
function to distribute conda environments. This is experimental, undocumented, and may change without notice. (:pr:`494`)- Add optional
dask_worker=
keyword toclient.run
functions that gets provided the worker or nanny object - Add
nanny=False
keyword toClient.run
, allowing for the execution of arbitrary functions on the nannies as well as normal workers
- Add optional
1.12.2
This release adds some new features and removes dead code
- Publish and share datasets on the scheduler between many clients (:pr:`453`). See :doc:`publish`.
- Launch tasks from other tasks (experimental) (:pr:`471`). See :doc:`task-launch`.
- Remove unused code, notably the
Center
object and older client functions (:pr:`478`) Executor()
andLocalCluster()
is now robust to Bokeh's absence (:pr:`481`)- Removed s3fs and boto3 from requirements. These have moved to Dask.
1.12.1
This release is largely a bugfix release, recovering from the previous large refactor.
- Fixes from previous refactor
- Ensure idempotence across clients
- Stress test losing scattered data permanently
- IPython fixes
- Add
start_ipython_scheduler
method to Executor - Add
%remote
magic for workers - Clean up code and tests
- Add
- Pool connects to maintain reuse and reduce number of open file handles
- Re-implement work stealing algorithm
- Support cancellation of tuple keys, such as occur in dask.arrays
- Start synchronizing against worker data that may be superfluous
- Improve bokeh plots styling
- Add memory plot tracking number of bytes
- Make the progress bars more compact and align colors
- Add workers/ page with workers table, stacks/processing plot, and memory
- Add this release notes document
1.12.0
This release was largely a refactoring release. Internals were changed significantly without many new features.
- Major refactor of the scheduler to use transitions system
- Tweak protocol to traverse down complex messages in search of large bytestrings
- Add dask-submit and dask-remote
- Refactor HDFS writing to align with changes in the dask library
- Executor reconnects to scheduler on broken connection or failed scheduler
- Support sklearn.external.joblib as well as normal joblib