/distributed

Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
Raw Blame History
849 lines (717 sloc) 43.1 KB

Changelog

1.24.2 - 2018-11-15

1.24.1 - 2018-11-09

1.24.0 - 2018-10-26

1.23.3 - 2018-10-05

1.23.2 - 2018-09-17

1.23.1 - 2018-09-06

1.23.0 - 2018-08-30

1.22.1 - 2018-08-03

1.22.0 - 2018-06-14

1.21.8 - 2018-05-03

1.21.7 - 2018-05-02

1.21.6 - 2018-04-06

1.21.5 - 2018-03-31

1.21.4 - 2018-03-21

1.21.3 - 2018-03-08

1.21.2 - 2018-03-05

1.21.1 - 2018-02-22

1.21.0 - 2018-02-09

1.20.2 - 2017-12-07

1.20.1 - 2017-11-26

1.20.0 - 2017-11-17

1.19.3 - 2017-10-16

1.19.2 - 2017-10-06

  • as_completed doesn't block on cancelled futures (:pr:`1436`)
  • Notify waiting threads/coroutines on cancellation (:pr:`1438`)
  • Set Future(inform=True) as default (:pr:`1437`)
  • Rename Scheduler.transition_story to story (:pr:`1445`)
  • Future uses default client by default (:pr:`1449`)
  • Add keys= keyword to Client.call_stack (:pr:`1446`)
  • Add get_current_task to worker (:pr:`1444`)
  • Ensure that Client remains asynchornous before ioloop starts (:pr:`1452`)
  • Remove "click for worker page" in bokeh plot (:pr:`1453`)
  • Add Client.current() (:pr:`1450`)
  • Clean handling of restart timeouts (:pr:`1442`)

1.19.1 - September 25th, 2017

1.19.0 - September 24th, 2017

  • Avoid storing messages in message log (:pr:`1361`)
  • fileConfig does not disable existing loggers (:pr:`1380`)
  • Offload upload_file disk I/O to separate thread (:pr:`1383`)
  • Add missing SSLContext (:pr:`1385`)
  • Collect worker thread information from sys._curent_frames (:pr:`1387`)
  • Add nanny timeout (:pr:`1395`)
  • Restart worker if memory use goes above 95% (:pr:`1397`)
  • Track workers memory use with psutil (:pr:`1398`)
  • Track scheduler delay times in workers (:pr:`1400`)
  • Add time slider to profile plot (:pr:`1403`)
  • Change memory-limit keyword to refer to maximum number of bytes (:pr:`1405`)
  • Add cancel(force=) keyword (:pr:`1408`)

1.18.2 - September 2nd, 2017

  • Silently pass on cancelled futures in as_completed (:pr:`1366`)
  • Fix unicode keys error in Python 2 (:pr:`1370`)
  • Support numeric worker names
  • Add dask-mpi executable (:pr:`1367`)

1.18.1 - August 25th, 2017

1.18.0 - July 8th, 2017

1.17.1 - June 14th, 2017

1.17.0 - June 9th, 2017

  • Reevaluate worker occupancy periodically during scheduler downtime (:pr:`1038`) (:pr:`1101`)
  • Add AioClient asyncio-compatible client API (:pr:`1029`) (:pr:`1092`) (:pr:`1099`)
  • Update Keras serializer (:pr:`1067`)
  • Support TLS/SSL connections for security (:pr:`866`) (:pr:`1034`)
  • Always create new worker directory when passed --local-directory (:pr:`1079`)
  • Support pre-scattering data when using joblib frontent (:pr:`1022`)
  • Make workers more robust to failure of sizeof function (:pr:`1108`) and writing to disk (:pr:`1096`)
  • Add is_empty and update methods to as_completed (:pr:`1113`)
  • Remove _get coroutine and replace with get(..., sync=False) (:pr:`1109`)
  • Improve API compatibility with async/await syntax (:pr:`1115`) (:pr:`1124`)
  • Add distributed Queues (:pr:`1117`) and shared Variables (:pr:`1128`) to enable inter-client coordination
  • Support direct client-to-worker scattering and gathering (:pr:`1130`) as well as performance enhancements when scattering data
  • Style improvements for bokeh web dashboards (:pr:`1126`) (:pr:`1141`) as well as a removal of the external bokeh process
  • HTML reprs for Future and Client objects (:pr:`1136`)
  • Support nested collections in client.compute (:pr:`1144`)
  • Use normal client API in asynchronous mode (:pr:`1152`)
  • Remove old distributed.collections submodule (:pr:`1153`)

1.16.3 - May 5th, 2017

  • Add bokeh template files to MANIFEST (:pr:`1063`)
  • Don't set worker_client.get as default get (:pr:`1061`)
  • Clean up logging on Client().shutdown() (:pr:`1055`)

1.16.2 - May 3rd, 2017

  • Support async with Client syntax (:pr:`1053`)
  • Use internal bokeh server for default diagnostics server (:pr:`1047`)
  • Improve styling of bokeh plots when empty (:pr:`1046`) (:pr:`1037`)
  • Support efficient serialization for sparse arrays (:pr:`1040`)
  • Prioritize newly arrived work in worker (:pr:`1035`)
  • Prescatter data with joblib backend (:pr:`1022`)
  • Make client.restart more robust to worker failure (:pr:`1018`)
  • Support preloading a module or script in dask-worker or dask-scheduler processes (:pr:`1016`)
  • Specify network interface in command line interface (:pr:`1007`)
  • Client.scatter supports a single element (:pr:`1003`)
  • Use blosc compression on all memoryviews passing through comms (:pr:`998`)
  • Add concurrent.futures-compatible Executor (:pr:`997`)
  • Add as_completed.batches method and return results (:pr:`994`) (:pr:`971`)
  • Allow worker_clients to optionally stay within the thread pool (:pr:`993`)
  • Add bytes-stored and tasks-processing diagnostic histograms (:pr:`990`)
  • Run supports non-msgpack-serializable results (:pr:`965`)

1.16.1 - March 22nd, 2017

  • Use inproc transport in LocalCluster (:pr:`919`)
  • Add structured and queryable cluster event logs (:pr:`922`)
  • Use connection pool for inter-worker communication (:pr:`935`)
  • Robustly shut down spawned worker processes at shutdown (:pr:`928`)
  • Worker death timeout (:pr:`940`)
  • More visual reporting of exceptions in progressbar (:pr:`941`)
  • Render disk and serialization events to task stream visual (:pr:`943`)
  • Support async for / await protocol (:pr:`952`)
  • Ensure random generators are re-seeded in worker processes (:pr:`953`)
  • Upload sourcecode as zip module (:pr:`886`)
  • Replay remote exceptions in local process (:pr:`894`)

1.16.0 - February 24th, 2017

  • First come first served priorities on client submissions (:pr:`840`)
  • Can specify Bokeh internal ports (:pr:`850`)
  • Allow stolen tasks to return from either worker (:pr:`853`), (:pr:`875`)
  • Add worker resource constraints during execution (:pr:`857`)
  • Send small data through Channels (:pr:`858`)
  • Better estimates for SciPy sparse matrix memory costs (:pr:`863`)
  • Avoid stealing long running tasks (:pr:`873`)
  • Maintain fortran ordering of NumPy arrays (:pr:`876`)
  • Add --scheduler-file keyword to dask-scheduler (:pr:`877`)
  • Add serializer for Keras models (:pr:`878`)
  • Support uploading modules from zip files (:pr:`886`)
  • Improve titles of Bokeh dashboards (:pr:`895`)

1.15.2 - January 27th, 2017

  • Fix a bug where arrays with large dtypes or shapes were being improperly compressed (:pr:`830` :pr:`832` :pr:`833`)
  • Extend as_completed to accept new futures during iteration (:pr:`829`)
  • Add --nohost keyword to dask-ssh startup utility (:pr:`827`)
  • Support scheduler shutdown of remote workers, useful for adaptive clusters (:pr: 811 :pr:`816` :pr:`821`)
  • Add Client.run_on_scheduler method for running debug functions on the scheduler (:pr:`808`)

1.15.1 - January 11th, 2017

  • Make compatibile with Bokeh 0.12.4 (:pr:`803`)
  • Avoid compressing arrays if not helpful (:pr:`777`)
  • Optimize inter-worker data transfer (:pr:`770`) (:pr:`790`)
  • Add --local-directory keyword to worker (:pr:`788`)
  • Enable workers to arrive to the cluster with their own data. Useful if a worker leaves and comes back (:pr:`785`)
  • Resolve thread safety bug when using local_client (:pr:`802`)
  • Resolve scheduling issues in worker (:pr:`804`)

1.15.0 - January 2nd, 2017

1.14.3 - November 13th, 2016

  • Remove custom Bokeh export tool that implicitly relied on nodejs (:pr:`655`)
  • Clean up scheduler logging (:pr:`657`)

1.14.2 - November 11th, 2016

1.14.0 - November 3rd, 2016

1.13.3 - October 15th, 2016

  • Schedulers can retire workers cleanly
  • Add Future.add_done_callback for concurrent.futures compatibility
  • Update web interface to be consistent with Bokeh 0.12.3
  • Close streams explicitly, avoiding race conditions and supporting more robust restarts on Windows.
  • Improved shuffled performance for dask.dataframe
  • Add adaptive allocation cluster manager
  • Reduce administrative overhead when dealing with many workers
  • dask-ssh --log-directory . no longer errors
  • Microperformance tuning for the scheduler

1.13.2

  • Revert dask_worker to use fork rather than subprocess by default
  • Scatter retains type information
  • Bokeh always uses subprocess rather than spawn

1.13.1

  • Fix critical Windows error with dask_worker executable

1.13.0

  • Rename Executor to Client (:pr:`492`)
  • Add --memory-limit option to dask-worker, enabling spill-to-disk behavior when running out of memory (:pr:`485`)
  • Add --pid-file option to dask-worker and --dask-scheduler (:pr:`496`)
  • Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (:pr:`494`)
  • Add workers= keyword argument to Client.compute and Client.persist, supporting location-restricted workloads with Dask collections (:pr:`484`)
  • Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (:pr:`494`)
    • Add optional dask_worker= keyword to client.run functions that gets provided the worker or nanny object
    • Add nanny=False keyword to Client.run, allowing for the execution of arbitrary functions on the nannies as well as normal workers

1.12.2

This release adds some new features and removes dead code

  • Publish and share datasets on the scheduler between many clients (:pr:`453`). See :doc:`publish`.
  • Launch tasks from other tasks (experimental) (:pr:`471`). See :doc:`task-launch`.
  • Remove unused code, notably the Center object and older client functions (:pr:`478`)
  • Executor() and LocalCluster() is now robust to Bokeh's absence (:pr:`481`)
  • Removed s3fs and boto3 from requirements. These have moved to Dask.

1.12.1

This release is largely a bugfix release, recovering from the previous large refactor.

  • Fixes from previous refactor
    • Ensure idempotence across clients
    • Stress test losing scattered data permanently
  • IPython fixes
    • Add start_ipython_scheduler method to Executor
    • Add %remote magic for workers
    • Clean up code and tests
  • Pool connects to maintain reuse and reduce number of open file handles
  • Re-implement work stealing algorithm
  • Support cancellation of tuple keys, such as occur in dask.arrays
  • Start synchronizing against worker data that may be superfluous
  • Improve bokeh plots styling
    • Add memory plot tracking number of bytes
    • Make the progress bars more compact and align colors
    • Add workers/ page with workers table, stacks/processing plot, and memory
  • Add this release notes document

1.12.0

This release was largely a refactoring release. Internals were changed significantly without many new features.

  • Major refactor of the scheduler to use transitions system
  • Tweak protocol to traverse down complex messages in search of large bytestrings
  • Add dask-submit and dask-remote
  • Refactor HDFS writing to align with changes in the dask library
  • Executor reconnects to scheduler on broken connection or failed scheduler
  • Support sklearn.external.joblib as well as normal joblib