Skip to content

/ distributed

Permalink
master
Switch branches/tags

distributed/docs/source/changelog.rst

Go to file
@jrbourbeau
jrbourbeau bump version to 2021.03.0
Latest commit 5ea74cd Mar 5, 2021 History
7 contributors

Users who have contributed to this file

2231 lines (1916 sloc) 124 KB
Raw Blame

Changelog

2021.03.0

Released on March 5, 2021

Note

This is the first release with support for Python 3.9 and the last release with support for Python 3.6

2021.02.0

Released on February 5, 2021

2021.01.1

Released on January 22, 2021

2021.01.0

Released on January 15, 2021

2020.12.0

Released on December 10, 2020

Highlights

  • Switched to CalVer for versioning scheme.
  • The scheduler can now receives Dask HighLevelGraph s instead of raw dictionary task graphs. This allows for a much more efficient communication of task graphs from the client to the scheduler.
  • Added support for using custom Layer-level annotations like priority, retries, etc. with the dask.annotations context manager.
  • Updated minimum supported version of Dask to 2020.12.0.
  • Added many type annotations and updates to allow for gradually Cythonizing the scheduler.

All changes

2.30.1 - 2020-11-03

2.30.0 - 2020-10-06

2.29.0 - 2020-10-02

2.28.0 - 2020-09-25

2.27.0 - 2020-09-18

2.26.0 - 2020-09-11

2.25.0 - 2020-08-28

2.24.0 - 2020-08-22

  • Move toolbar to above and fix y axis (#4043) Julia Signell
  • Make behavior clearer for how to get worker dashboard (#4047) Julia Signell
  • Worker dashboard clean up (#4046) Julia Signell
  • Add a default argument to the datasets and a possibility to override datasets (#4052) Nils Braun
  • Discover HTTP endpoints (#3744) Martin Durant

2.23.0 - 2020-08-14

2.22.0 - 2020-07-31

2.21.0 - 2020-07-17

2.20.0 - 2020-07-02

2.19.0 - 2020-06-19

2.18.0 - 2020-06-05

2.17.0 - 2020-05-26

2.16.0 - 2020-05-08

2.15.2 - 2020-05-01

2.15.1 - 2020-04-28

2.15.0 - 2020-04-24

2.14.0 - 2020-04-03

2.13.0 - 2020-03-25

2.12.0 - 2020-03-06

2.11.0 - 2020-02-19

2.10.0 - 2020-01-28

2.9.3 - 2020-01-17

2.9.2 - 2020-01-16

2.9.1 - 2019-12-27

2.9.0 - 2019-12-06

2.8.1 - 2019-11-22

2.8.0 - 2019-11-14

2.7.0 - 2019-11-08

This release drops support for Python 3.5

2.6.0 - 2019-10-15

2.5.2 - 2019-10-04

2.5.1 - 2019-09-27

2.5.0 - 2019-09-27

2.4.0 - 2019-09-13

2.3.2 - 2019-08-23

2.3.1 - 2019-08-22

2.3.0 - 2019-08-16

2.2.0 - 2019-07-31

2.1.0 - 2019-07-08

2.0.1 - 2019-06-26

We neglected to include python_requires= in our setup.py file, resulting in confusion for Python 2 users who erroneously get packages for 2.0.0. This is fixed in 2.0.1 and we have removed the 2.0.0 files from PyPI.

2.0.0 - 2019-06-25

1.28.1 - 2019-05-13

This is a small bugfix release due to a config change upstream.

1.28.0 - 2019-05-08

1.27.1 - 2019-04-29

1.27.0 - 2019-04-12

1.26.1 - 2019-03-29

1.26.0 - 2019-02-25

1.25.3 - 2019-01-31

1.25.2 - 2019-01-04

1.25.1 - 2018-12-15

1.25.0 - 2018-11-28

1.24.2 - 2018-11-15

1.24.1 - 2018-11-09

1.24.0 - 2018-10-26

1.23.3 - 2018-10-05

1.23.2 - 2018-09-17

1.23.1 - 2018-09-06

1.23.0 - 2018-08-30

1.22.1 - 2018-08-03

1.22.0 - 2018-06-14

1.21.8 - 2018-05-03

1.21.7 - 2018-05-02

1.21.6 - 2018-04-06

1.21.5 - 2018-03-31

1.21.4 - 2018-03-21

1.21.3 - 2018-03-08

1.21.2 - 2018-03-05

1.21.1 - 2018-02-22

1.21.0 - 2018-02-09

1.20.2 - 2017-12-07

1.20.1 - 2017-11-26

1.20.0 - 2017-11-17

1.19.3 - 2017-10-16

1.19.2 - 2017-10-06

  • as_completed doesn't block on cancelled futures (:pr:`1436`)
  • Notify waiting threads/coroutines on cancellation (:pr:`1438`)
  • Set Future(inform=True) as default (:pr:`1437`)
  • Rename Scheduler.transition_story to story (:pr:`1445`)
  • Future uses default client by default (:pr:`1449`)
  • Add keys= keyword to Client.call_stack (:pr:`1446`)
  • Add get_current_task to worker (:pr:`1444`)
  • Ensure that Client remains asynchornous before ioloop starts (:pr:`1452`)
  • Remove "click for worker page" in bokeh plot (:pr:`1453`)
  • Add Client.current() (:pr:`1450`)
  • Clean handling of restart timeouts (:pr:`1442`)

1.19.1 - September 25th, 2017

1.19.0 - September 24th, 2017

  • Avoid storing messages in message log (:pr:`1361`)
  • fileConfig does not disable existing loggers (:pr:`1380`)
  • Offload upload_file disk I/O to separate thread (:pr:`1383`)
  • Add missing SSLContext (:pr:`1385`)
  • Collect worker thread information from sys._curent_frames (:pr:`1387`)
  • Add nanny timeout (:pr:`1395`)
  • Restart worker if memory use goes above 95% (:pr:`1397`)
  • Track workers memory use with psutil (:pr:`1398`)
  • Track scheduler delay times in workers (:pr:`1400`)
  • Add time slider to profile plot (:pr:`1403`)
  • Change memory-limit keyword to refer to maximum number of bytes (:pr:`1405`)
  • Add cancel(force=) keyword (:pr:`1408`)

1.18.2 - September 2nd, 2017

  • Silently pass on cancelled futures in as_completed (:pr:`1366`)
  • Fix unicode keys error in Python 2 (:pr:`1370`)
  • Support numeric worker names
  • Add dask-mpi executable (:pr:`1367`)

1.18.1 - August 25th, 2017

1.18.0 - July 8th, 2017

1.17.1 - June 14th, 2017

1.17.0 - June 9th, 2017

  • Reevaluate worker occupancy periodically during scheduler downtime (:pr:`1038`) (:pr:`1101`)
  • Add AioClient asyncio-compatible client API (:pr:`1029`) (:pr:`1092`) (:pr:`1099`)
  • Update Keras serializer (:pr:`1067`)
  • Support TLS/SSL connections for security (:pr:`866`) (:pr:`1034`)
  • Always create new worker directory when passed --local-directory (:pr:`1079`)
  • Support pre-scattering data when using joblib frontent (:pr:`1022`)
  • Make workers more robust to failure of sizeof function (:pr:`1108`) and writing to disk (:pr:`1096`)
  • Add is_empty and update methods to as_completed (:pr:`1113`)
  • Remove _get coroutine and replace with get(..., sync=False) (:pr:`1109`)
  • Improve API compatibility with async/await syntax (:pr:`1115`) (:pr:`1124`)
  • Add distributed Queues (:pr:`1117`) and shared Variables (:pr:`1128`) to enable inter-client coordination
  • Support direct client-to-worker scattering and gathering (:pr:`1130`) as well as performance enhancements when scattering data
  • Style improvements for bokeh web dashboards (:pr:`1126`) (:pr:`1141`) as well as a removal of the external bokeh process
  • HTML reprs for Future and Client objects (:pr:`1136`)
  • Support nested collections in client.compute (:pr:`1144`)
  • Use normal client API in asynchronous mode (:pr:`1152`)
  • Remove old distributed.collections submodule (:pr:`1153`)

1.16.3 - May 5th, 2017

  • Add bokeh template files to MANIFEST (:pr:`1063`)
  • Don't set worker_client.get as default get (:pr:`1061`)
  • Clean up logging on Client().shutdown() (:pr:`1055`)

1.16.2 - May 3rd, 2017

  • Support async with Client syntax (:pr:`1053`)
  • Use internal bokeh server for default diagnostics server (:pr:`1047`)
  • Improve styling of bokeh plots when empty (:pr:`1046`) (:pr:`1037`)
  • Support efficient serialization for sparse arrays (:pr:`1040`)
  • Prioritize newly arrived work in worker (:pr:`1035`)
  • Prescatter data with joblib backend (:pr:`1022`)
  • Make client.restart more robust to worker failure (:pr:`1018`)
  • Support preloading a module or script in dask-worker or dask-scheduler processes (:pr:`1016`)
  • Specify network interface in command line interface (:pr:`1007`)
  • Client.scatter supports a single element (:pr:`1003`)
  • Use blosc compression on all memoryviews passing through comms (:pr:`998`)
  • Add concurrent.futures-compatible Executor (:pr:`997`)
  • Add as_completed.batches method and return results (:pr:`994`) (:pr:`971`)
  • Allow worker_clients to optionally stay within the thread pool (:pr:`993`)
  • Add bytes-stored and tasks-processing diagnostic histograms (:pr:`990`)
  • Run supports non-msgpack-serializable results (:pr:`965`)

1.16.1 - March 22nd, 2017

  • Use inproc transport in LocalCluster (:pr:`919`)
  • Add structured and queryable cluster event logs (:pr:`922`)
  • Use connection pool for inter-worker communication (:pr:`935`)
  • Robustly shut down spawned worker processes at shutdown (:pr:`928`)
  • Worker death timeout (:pr:`940`)
  • More visual reporting of exceptions in progressbar (:pr:`941`)
  • Render disk and serialization events to task stream visual (:pr:`943`)
  • Support async for / await protocol (:pr:`952`)
  • Ensure random generators are re-seeded in worker processes (:pr:`953`)
  • Upload sourcecode as zip module (:pr:`886`)
  • Replay remote exceptions in local process (:pr:`894`)

1.16.0 - February 24th, 2017

  • First come first served priorities on client submissions (:pr:`840`)
  • Can specify Bokeh internal ports (:pr:`850`)
  • Allow stolen tasks to return from either worker (:pr:`853`), (:pr:`875`)
  • Add worker resource constraints during execution (:pr:`857`)
  • Send small data through Channels (:pr:`858`)
  • Better estimates for SciPy sparse matrix memory costs (:pr:`863`)
  • Avoid stealing long running tasks (:pr:`873`)
  • Maintain fortran ordering of NumPy arrays (:pr:`876`)
  • Add --scheduler-file keyword to dask-scheduler (:pr:`877`)
  • Add serializer for Keras models (:pr:`878`)
  • Support uploading modules from zip files (:pr:`886`)
  • Improve titles of Bokeh dashboards (:pr:`895`)

1.15.2 - January 27th, 2017

  • Fix a bug where arrays with large dtypes or shapes were being improperly compressed (:pr:`830` :pr:`832` :pr:`833`)
  • Extend as_completed to accept new futures during iteration (:pr:`829`)
  • Add --nohost keyword to dask-ssh startup utility (:pr:`827`)
  • Support scheduler shutdown of remote workers, useful for adaptive clusters (:pr: 811 :pr:`816` :pr:`821`)
  • Add Client.run_on_scheduler method for running debug functions on the scheduler (:pr:`808`)

1.15.1 - January 11th, 2017

  • Make compatibile with Bokeh 0.12.4 (:pr:`803`)
  • Avoid compressing arrays if not helpful (:pr:`777`)
  • Optimize inter-worker data transfer (:pr:`770`) (:pr:`790`)
  • Add --local-directory keyword to worker (:pr:`788`)
  • Enable workers to arrive to the cluster with their own data. Useful if a worker leaves and comes back (:pr:`785`)
  • Resolve thread safety bug when using local_client (:pr:`802`)
  • Resolve scheduling issues in worker (:pr:`804`)

1.15.0 - January 2nd, 2017

1.14.3 - November 13th, 2016

  • Remove custom Bokeh export tool that implicitly relied on nodejs (:pr:`655`)
  • Clean up scheduler logging (:pr:`657`)

1.14.2 - November 11th, 2016

1.14.0 - November 3rd, 2016

1.13.3 - October 15th, 2016

  • Schedulers can retire workers cleanly
  • Add Future.add_done_callback for concurrent.futures compatibility
  • Update web interface to be consistent with Bokeh 0.12.3
  • Close streams explicitly, avoiding race conditions and supporting more robust restarts on Windows.
  • Improved shuffled performance for dask.dataframe
  • Add adaptive allocation cluster manager
  • Reduce administrative overhead when dealing with many workers
  • dask-ssh --log-directory . no longer errors
  • Microperformance tuning for the scheduler

1.13.2

  • Revert dask_worker to use fork rather than subprocess by default
  • Scatter retains type information
  • Bokeh always uses subprocess rather than spawn

1.13.1

  • Fix critical Windows error with dask_worker executable

1.13.0

  • Rename Executor to Client (:pr:`492`)
  • Add --memory-limit option to dask-worker, enabling spill-to-disk behavior when running out of memory (:pr:`485`)
  • Add --pid-file option to dask-worker and --dask-scheduler (:pr:`496`)
  • Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (:pr:`494`)
  • Add workers= keyword argument to Client.compute and Client.persist, supporting location-restricted workloads with Dask collections (:pr:`484`)
  • Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (:pr:`494`)
    • Add optional dask_worker= keyword to client.run functions that gets provided the worker or nanny object
    • Add nanny=False keyword to Client.run, allowing for the execution of arbitrary functions on the nannies as well as normal workers

1.12.2

This release adds some new features and removes dead code

  • Publish and share datasets on the scheduler between many clients (:pr:`453`). See :doc:`publish`.
  • Launch tasks from other tasks (experimental) (:pr:`471`). See :doc:`task-launch`.
  • Remove unused code, notably the Center object and older client functions (:pr:`478`)
  • Executor() and LocalCluster() is now robust to Bokeh's absence (:pr:`481`)
  • Removed s3fs and boto3 from requirements. These have moved to Dask.

1.12.1

This release is largely a bugfix release, recovering from the previous large refactor.

  • Fixes from previous refactor
    • Ensure idempotence across clients
    • Stress test losing scattered data permanently
  • IPython fixes
    • Add start_ipython_scheduler method to Executor
    • Add %remote magic for workers
    • Clean up code and tests
  • Pool connects to maintain reuse and reduce number of open file handles
  • Re-implement work stealing algorithm
  • Support cancellation of tuple keys, such as occur in dask.arrays
  • Start synchronizing against worker data that may be superfluous
  • Improve bokeh plots styling
    • Add memory plot tracking number of bytes
    • Make the progress bars more compact and align colors
    • Add workers/ page with workers table, stacks/processing plot, and memory
  • Add this release notes document

1.12.0

This release was largely a refactoring release. Internals were changed significantly without many new features.

  • Major refactor of the scheduler to use transitions system
  • Tweak protocol to traverse down complex messages in search of large bytestrings
  • Add dask-submit and dask-remote
  • Refactor HDFS writing to align with changes in the dask library
  • Executor reconnects to scheduler on broken connection or failed scheduler
  • Support sklearn.external.joblib as well as normal joblib